Nov 02 11:53:37.610592 ip-10-0-133-239 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 02 11:53:37.610601 ip-10-0-133-239 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 02 11:53:37.610608 ip-10-0-133-239 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 02 11:53:37.610828 ip-10-0-133-239 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 02 11:53:47.816601 ip-10-0-133-239 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 02 11:53:47.816620 ip-10-0-133-239 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3d55430b42654e619b3431b1587b356a --
Nov 02 11:55:31.643268 ip-10-0-133-239 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 11:55:32.077475 ip-10-0-133-239 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 11:55:32.077475 ip-10-0-133-239 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 11:55:32.077475 ip-10-0-133-239 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 11:55:32.077475 ip-10-0-133-239 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 11:55:32.084331 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084250    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 11:55:32.084331 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084327    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 11:55:32.084331 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084331    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 11:55:32.084331 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084335    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 11:55:32.084331 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084338    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084343    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084346    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084348    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084351    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084354    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084356    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084359    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084362    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084365    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084369    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084372    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084375    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084377    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084380    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084383    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084385    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084388    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084390    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 11:55:32.084516 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084393    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084396    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084398    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084401    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084403    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084406    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084409    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084412    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084415    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084418    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084421    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084424    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084427    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084429    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084432    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084435    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084438    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084441    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084443    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084445    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 11:55:32.084950 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084448    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084451    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084454    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084457    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084459    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084462    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084465    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084470    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084472    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084475    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084478    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084481    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084484    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084486    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084489    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084492    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084494    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084497    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084499    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084503    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 11:55:32.085452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084506    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084508    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084511    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084514    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084516    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084519    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084523    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084526    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084529    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084531    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084534    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084537    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084539    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084542    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084546    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084550    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084558    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084562    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084565    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 11:55:32.085921 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084568    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084571    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084574    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084578    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084581    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084584    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084586    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084589    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084599    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084602    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084605    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084607    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084610    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084613    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084616    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084618    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084621    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084625    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084629    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 11:55:32.086385 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084633    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084636    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084639    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084641    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.084646    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085461    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085469    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085473    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085476    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085479    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085481    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085484    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085487    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085491    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085496    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085500    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085503    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085506    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085509    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 11:55:32.086834 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085511    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085514    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085517    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085519    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085522    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085525    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085528    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085531    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085533    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085536    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085539    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085541    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085544    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085548    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085552    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085556    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085559    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085563    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085565    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 11:55:32.087310 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085569    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085571    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085575    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085578    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085581    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085584    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085587    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085589    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085592    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085594    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085598    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085601    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085603    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085606    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085609    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085611    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085614    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085617    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085619    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085622    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 11:55:32.087759 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085624    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085627    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085630    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085632    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085636    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085639    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085641    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085644    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085646    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085649    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085652    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085654    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085657    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085660    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085662    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085665    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085668    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085670    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085673    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 11:55:32.088249 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085676    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085679    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085681    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085684    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085687    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085689    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085692    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085694    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085697    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085699    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085702    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085704    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085707    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085710    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085712    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085716    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085719    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085722    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085724    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085727    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 11:55:32.088702 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085729    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085732    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085734    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085737    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085740    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085745    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085747    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085750    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085752    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085755    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085759    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085761    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085764    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.085766    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086346    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086353    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086357    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086360    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086362    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086365    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 11:55:32.089209 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086368    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086371    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086373    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086376    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086379    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086381    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086384    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086387    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086390    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086393    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086396    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086399    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086402    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086404    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086407    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086409    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086412    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086415    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086418    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086421    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 11:55:32.089663 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086423    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086426    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086428    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086431    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086433    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086436    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086438    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086441    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086443    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086446    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086449    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086452    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086455    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086457    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086460    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086462    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086465    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086468    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086471    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086474    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 11:55:32.090141 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086477    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086480    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086482    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086484    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086487    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086489    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086492    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086494    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086497    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086500    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086503    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086505    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086508    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086510    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086513    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086515    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086518    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086521    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086524    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 11:55:32.090599 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086526    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086529    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086531    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086534    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086538    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086542    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086545    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086547    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086550    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086553    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086555    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086557    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086560    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086563    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086565    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086567    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086570    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086573    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086577    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 11:55:32.091071 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086581    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086584    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086587    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086590    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086593    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086595    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086598    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086600    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086603    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086605    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086608    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086610    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086613    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086615    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086618    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086620    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086623    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086626    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086629    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 11:55:32.091512 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086632    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086635    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.086637    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087431    2577 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087441    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087447    2577 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087452    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087456    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087461    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087469    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087474    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087477    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087480    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087484    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087487    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087490    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087493    2577 flags.go:64] FLAG: --cgroup-root=""
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087496    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087499    2577 flags.go:64] FLAG: --client-ca-file=""
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087502    2577 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087505    2577 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087510    2577 flags.go:64] FLAG: --cluster-domain=""
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087513    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 11:55:32.092000 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087517    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087520    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087523    2577 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087527    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087530    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087534    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087537    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087540    2577 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087544    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087547    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087550    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087553    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087558    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087561    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087564    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087567    2577 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087572    2577 flags.go:64] FLAG: --enable-server="true"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087575    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087580    2577 flags.go:64] FLAG: --event-burst="100"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087583    2577 flags.go:64] FLAG: --event-qps="50"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087586    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087589    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087592    2577 flags.go:64] FLAG: --eviction-hard=""
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087596    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 11:55:32.092613 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087599    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087602    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087605    2577 flags.go:64] FLAG: --eviction-soft=""
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087608    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087611    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087614    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087617    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087620    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087623    2577 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087626    2577 flags.go:64] FLAG: --feature-gates=""
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087635    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087638    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087641    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087644    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087647    2577 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087650    2577 flags.go:64] FLAG: --help="false"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087653    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087657    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087660    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087663    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087666    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087670    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087673    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 11:55:32.093239 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087676    2577 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087681    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087684    2577 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087686    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087690    2577 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087693    2577 flags.go:64] FLAG: --kube-reserved=""
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087696    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087699    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087702    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087705    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087708    2577 flags.go:64] FLAG: --lock-file=""
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087711    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087714    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087717    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087722    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087725    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087728    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087733    2577 flags.go:64] FLAG: --logging-format="text"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087736    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087739    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087742    2577 flags.go:64] FLAG: --manifest-url=""
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087746    2577 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087750    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087753    2577 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087757    2577 flags.go:64] FLAG: --max-pods="110"
Nov 02 11:55:32.093828 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087760    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087763    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087767    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087770    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087773    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087776    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087779    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087787    2577 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087791    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087794    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087797    2577 flags.go:64] FLAG: --pod-cidr=""
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087800    2577 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087803    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087807    2577 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087809    2577 flags.go:64] FLAG: --port="10250"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087813    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087815    2577 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-096a8732a36243750"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087819    2577 flags.go:64] FLAG: --qos-reserved=""
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087822    2577 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087825    2577 flags.go:64] FLAG: --register-node="true"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087828    2577 flags.go:64] FLAG: --register-with-taints=""
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087832    2577 flags.go:64] FLAG: --registry-burst="10"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087835    2577 flags.go:64] FLAG: --registry-qps="5"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087838    2577 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087842    2577 flags.go:64] FLAG: --reserved-memory=""
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087846    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 11:55:32.094440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087849    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087852    2577 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087855    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087858    2577 flags.go:64] FLAG: --runonce="false"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087861    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087864    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087867    2577 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087870    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087873    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087876    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087880    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087883    2577 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087886    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087889    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087893    2577 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087896    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087899    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087903    2577 flags.go:64] FLAG: --system-cgroups=""
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087905    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087911    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087914    2577 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087916    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087921    2577 flags.go:64] FLAG: --tls-min-version=""
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087924    2577 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087927    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 11:55:32.095075 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087930    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087933    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087936    2577 flags.go:64] FLAG: --v="2"
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087940    2577 flags.go:64] FLAG: --version="false"
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087944    2577 flags.go:64] FLAG: --vmodule=""
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087949    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.087953    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088109    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088114    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088116    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088119    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088122    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088125    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088127    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088130    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088133    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088136    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088139    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088142    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088145    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088148    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 11:55:32.095674 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088152    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088155    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088158    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088160    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088163    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088166    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088168    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088171    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088174    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088176    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088179    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088182    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088184    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088187    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088190    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088193    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088196    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088199    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088202    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088205    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 11:55:32.096233 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088207    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088210    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088213    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088215    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088219    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088223    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088226    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088230    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088233    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088236    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088239    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088242    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088245    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088248    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088251    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088254    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088256    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088260    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088262    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088265    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 11:55:32.096714 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088267    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088271    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088273    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088276    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088279    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088282    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088284    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088287    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088291    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088294    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088297    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088299    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088302    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088305    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088308    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088311    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088313    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088316    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088318    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088321    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 11:55:32.097254 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088323    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088326    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088329    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088332    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088336    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088339    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088341    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088344    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088347    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088349    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088352    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088354    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088357    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088360    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088363    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088365    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088368    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088370    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088373    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088375    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 11:55:32.097716 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088379    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 11:55:32.098212 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088382    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 11:55:32.098212 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088384    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 11:55:32.098212 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088387    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 11:55:32.098212 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088389    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 11:55:32.098212 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088392    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 11:55:32.098212 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088395    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 11:55:32.098212 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088398    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 11:55:32.098212 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088402    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 11:55:32.098212 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088405    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 11:55:32.098212 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088408    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 11:55:32.098212 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088411    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 11:55:32.098212 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.088436    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 11:55:32.098212 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088475    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 11:55:32.098212 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088479    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 11:55:32.098212 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088483    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088487    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088490    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088493    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088495    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088498    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088501    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088504    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088506    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088522    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088527    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088530    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088533    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088537    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088539    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088543    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088545    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088548    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088552    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088555    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 11:55:32.098563 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088558    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088560    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088563    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088566    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088569    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088571    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088574    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088576    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088579    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088582    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088584    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088587    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088589    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088596    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088600    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088603    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088605    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088608    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088611    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088613    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 11:55:32.099052 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088616    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088619    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088621    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088624    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088626    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088629    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088632    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088634    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088637    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088639    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088643    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088646    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088649    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088651    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088654    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088657    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088661    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088665    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088668    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 11:55:32.099536 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088670    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088673    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088676    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088678    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088681    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088684    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088689    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088692    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088695    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088697    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088700    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088703    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088706    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088708    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088711    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088714    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088717    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088719    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088722    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 11:55:32.099994 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088724    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088727    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088730    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088732    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088735    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088738    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088741    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088743    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088746    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088748    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088751    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088754    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088756    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088759    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088761    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088764    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088766    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088769    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088771    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088775    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 11:55:32.100429 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088777    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088782    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088785    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088788    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088792    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.088795    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.095345    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.095360    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.095376    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.095381    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095436    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095441    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095445    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095449    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095452    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095455    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095457    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095460    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095463    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 11:55:32.100889 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095465    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095468    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095471    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095474    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095476    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095479    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095482    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095485    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095487    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095490    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095493    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095495    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095498    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095501    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095504    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095507    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095510    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095513    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095515    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 11:55:32.101371 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095518    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095522    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095525    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095528    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095531    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095534    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095536    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095539    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095541    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095545    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095549    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095552    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095555    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095557    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095560    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095563    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095565    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095569    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095573    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 11:55:32.101833 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095576    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095578    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095581    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095583    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095586    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095589    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095591    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095594    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095596    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095600    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095603    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095606    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095608    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095611    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095614    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095617    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095620    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095623    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095625    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095628    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 11:55:32.102328 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095630    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095633    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095636    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095638    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095641    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095643    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095646    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095648    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095651    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095654    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095656    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095659    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095661    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095664    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095666    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095669    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095672    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095674    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095677    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095680    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 11:55:32.102784 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095682    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095685    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095691    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095694    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095697    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095699    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095703    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095706    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095708    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095711    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095713    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095716    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095718    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095721    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095724    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095726    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095729    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095731    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 11:55:32.103393 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095734    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 11:55:32.103817 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.095761    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 11:55:32.103817 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095931    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 11:55:32.103817 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095936    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 11:55:32.103817 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095939    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 11:55:32.103817 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095942    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 11:55:32.103817 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095945    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 11:55:32.103817 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095948    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 11:55:32.103817 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095950    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 11:55:32.103817 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095958    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 11:55:32.103817 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095961    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 11:55:32.103817 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095964    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 11:55:32.103817 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095966    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 11:55:32.103817 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095969    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 11:55:32.103817 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095992    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 11:55:32.103817 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095996    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 11:55:32.103817 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.095999    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096008    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096011    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096013    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096016    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096019    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096022    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096024    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096027    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096030    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096032    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096035    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096038    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096040    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096043    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096046    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096048    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096051    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096053    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096056    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 11:55:32.104270 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096059    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096061    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096064    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096066    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096069    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096072    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096074    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096077    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096094    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096097    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096101    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096104    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096108    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096112    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096121    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096125    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096128    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096130    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096133    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 11:55:32.104743 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096136    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096138    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096141    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096143    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096146    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096148    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096151    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096153    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096156    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096158    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096161    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096163    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096166    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096168    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096171    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096174    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096176    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096179    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096181    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096184    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 11:55:32.105223 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096186    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096189    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096191    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096194    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096196    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096199    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096201    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096204    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096214    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096217    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096220    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096222    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096225    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096229    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096232    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096235    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096238    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096241    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 11:55:32.105706 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096244    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 11:55:32.106143 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096246    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 11:55:32.106143 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096249    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 11:55:32.106143 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096251    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 11:55:32.106143 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096254    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 11:55:32.106143 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096257    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 11:55:32.106143 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096259    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 11:55:32.106143 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096262    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 11:55:32.106143 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096265    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 11:55:32.106143 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096267    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 11:55:32.106143 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096270    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 11:55:32.106143 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096272    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 11:55:32.106143 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096275    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 11:55:32.106143 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096278    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 11:55:32.106143 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096280    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 11:55:32.106143 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.096305    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 11:55:32.106143 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096352    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096357    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096360    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096363    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096365    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096368    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096370    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096378    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096381    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096385    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096387    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096390    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096393    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096395    2577 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096398    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096401    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096403    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096406    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096408    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096411    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 11:55:32.106518 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096414    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096417    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096419    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096421    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096424    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096427    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096429    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096431    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096434    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096436    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096439    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096441    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096444    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096446    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096449    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096452    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096454    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096457    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096459    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096461    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 11:55:32.107000 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096469    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096472    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096475    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096477    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096480    2577 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096483    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096485    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096488    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096490    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096493    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096496    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096499    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096501    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096504    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096506    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096509    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096512    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096514    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096517    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096519    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 11:55:32.107459 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096522    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096524    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096527    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096531    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096535    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096538    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096540    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096543    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096546    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096548    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096550    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096553    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096555    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096563    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096566    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096569    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096571    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096574    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096576    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 11:55:32.107927 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096579    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096581    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096584    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096586    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096589    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096591    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096594    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096596    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096599    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096601    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096604    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096606    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096609    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096611    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096614    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096616    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096619    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096621    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096624    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 11:55:32.108391 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096627    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 11:55:32.108820 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096629    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 11:55:32.108820 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096632    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 11:55:32.108820 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096634    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 11:55:32.108820 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096637    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 11:55:32.108820 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096641    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 11:55:32.108820 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096644    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 11:55:32.108820 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:32.096647    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 11:55:32.108820 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.096700    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 11:55:32.108820 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.098654    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 11:55:32.108820 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.099532    2577 server.go:1014] "Starting client certificate rotation"
Nov 02 11:55:32.108820 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.099658    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 11:55:32.108820 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.099693    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 11:55:32.108820 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.101709    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 11:55:32.108820 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.103144    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 11:55:32.116128 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.116112    2577 log.go:25] "Validated CRI v1 runtime API"
Nov 02 11:55:32.119154 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.119140    2577 log.go:25] "Validated CRI v1 image API"
Nov 02 11:55:32.119876 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.119851    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 11:55:32.124422 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.124406    2577 fs.go:135] Filesystem UUIDs: map[1553d9df-aa57-4184-9f7d-46e8ac04b35b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c7db4a47-c6bf-4840-a0f6-49dbdb8520ae:/dev/nvme0n1p3]
Nov 02 11:55:32.124480 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.124422    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 11:55:32.130106 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.129995    2577 manager.go:217] Machine: {Timestamp:2026-11-02 11:55:32.127916046 +0000 UTC m=+0.348120792 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2993780 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d58a7fbab2e509e516dd25245ed50 SystemUUID:ec2d58a7-fbab-2e50-9e51-6dd25245ed50 BootID:3d55430b-4265-4e61-9b34-31b1587b356a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:01:48:5d:6c:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:01:48:5d:6c:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:58:1a:e2:45:d7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 11:55:32.130106 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.130105    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 11:55:32.130200 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.130176    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 11:55:32.131253 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.131225    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 11:55:32.131398 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.131254    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-239.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 11:55:32.132008 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.131997    2577 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 11:55:32.132068 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.132012    2577 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 11:55:32.132068 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.132051    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 11:55:32.132147 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.132071    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 11:55:32.132812 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.132800    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 11:55:32.136347 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.136334    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 11:55:32.136488 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.136477    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 11:55:32.137194 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.137184    2577 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 11:55:32.137245 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.137201    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 11:55:32.137245 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.137220    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 11:55:32.137245 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.137233    2577 kubelet.go:405] "Adding apiserver pod source"
Nov 02 11:55:32.137368 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.137247    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 11:55:32.137565 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.137551    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 11:55:32.141669 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.141653    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 11:55:32.146683 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.146668    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 11:55:32.147519 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.147505    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 11:55:32.147592 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.147545    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 11:55:32.147743 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.147730    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 11:55:32.147800 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.147754    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 11:55:32.147800 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.147770    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 11:55:32.147800 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.147785    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 11:55:32.147800 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.147794    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 11:55:32.147907 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.147803    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 11:55:32.148730 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.148720    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 11:55:32.148790 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.148734    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 11:55:32.148790 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.148748    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 11:55:32.148790 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.148757    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 11:55:32.148790 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.148769    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 11:55:32.148790 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.148784    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 11:55:32.149360 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.149349    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 11:55:32.149409 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.149364    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 11:55:32.153108 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.153093    2577 server.go:1263] "Started kubelet"
Nov 02 11:55:32.153250 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.153175    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 11:55:32.153382 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.153369    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 11:55:32.153457 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.153309    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 11:55:32.153764 ip-10-0-133-239 systemd[1]: Started Kubernetes Kubelet.
Nov 02 11:55:32.154518 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.154462    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 11:55:32.158161 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.157908    2577 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 11:55:32.160446 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:32.160423    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 11:55:32.162312 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.162295    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 11:55:32.163423 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.163407    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 11:55:32.163599 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:32.163561    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-239.us-west-1.compute.internal\" not found"
Nov 02 11:55:32.164128 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.164109    2577 factory.go:55] Registering systemd factory
Nov 02 11:55:32.164128 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.164128    2577 factory.go:223] Registration of the systemd container factory successfully
Nov 02 11:55:32.164354 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.164340    2577 factory.go:153] Registering CRI-O factory
Nov 02 11:55:32.164395 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.164356    2577 factory.go:223] Registration of the crio container factory successfully
Nov 02 11:55:32.164427 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.164396    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 11:55:32.164427 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.164409    2577 factory.go:103] Registering Raw factory
Nov 02 11:55:32.164427 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.164419    2577 manager.go:1199] Started watching for new ooms in manager
Nov 02 11:55:32.164834 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.164822    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 11:55:32.164867 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.164838    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 11:55:32.164896 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.164824    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 11:55:32.164938 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.164928    2577 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 11:55:32.164967 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.164939    2577 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 11:55:32.165240 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.165216    2577 manager.go:319] Starting recovery of all containers
Nov 02 11:55:32.175178 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.175160    2577 manager.go:324] Recovery completed
Nov 02 11:55:32.179543 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.179530    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 11:55:32.181136 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.181116    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 11:55:32.181222 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.181150    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 11:55:32.181222 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.181161    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 11:55:32.181716 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.181701    2577 cpu_manager.go:226] "Starting" policy="none"
Nov 02 11:55:32.181716 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.181711    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 11:55:32.181804 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.181712    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 11:55:32.181804 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.181753    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 11:55:32.182360 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.182345    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 11:55:32.182427 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.182378    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 11:55:32.182427 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.182394    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 11:55:32.183765 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.183753    2577 policy_none.go:50] "Start"
Nov 02 11:55:32.183838 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.183769    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 11:55:32.183838 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.183781    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 11:55:32.185783 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.185771    2577 policy_none.go:44] "Start"
Nov 02 11:55:32.227727 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.227592    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 11:55:32.233114 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.233102    2577 manager.go:341] "Starting Device Plugin manager"
Nov 02 11:55:32.233631 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:32.233618    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 11:55:32.233631 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.233631    2577 server.go:87] "Starting device plugin registration server"
Nov 02 11:55:32.233824 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.233814    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 11:55:32.233867 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.233826    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 11:55:32.233955 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.233941    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 11:55:32.234050 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.234039    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 11:55:32.234050 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.234050    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 11:55:32.234448 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:32.234429    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 11:55:32.234531 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:32.234470    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-239.us-west-1.compute.internal\" not found"
Nov 02 11:55:32.297871 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.297852    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-239.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 11:55:32.298779 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:32.297838    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-239.us-west-1.compute.internal.18e4385ef8d1df0b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-239.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-239.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 11:55:32.152885003 +0000 UTC m=+0.373089749,LastTimestamp:2026-11-02 11:55:32.152885003 +0000 UTC m=+0.373089749,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-239.us-west-1.compute.internal,}"
Nov 02 11:55:32.334206 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.334163    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 11:55:32.354420 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.335012    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 11:55:32.354420 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.335056    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 11:55:32.354420 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.335071    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 11:55:32.354420 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.335101    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.354420 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:32.346365    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-239.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 11:55:32.354420 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.351047    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 11:55:32.354420 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.352230    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 11:55:32.354420 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.352254    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 11:55:32.354420 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.352275    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 11:55:32.354420 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:32.352312    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 11:55:32.378372 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.378352    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 11:55:32.378448 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.378403    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cl788"
Nov 02 11:55:32.389529 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.389508    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cl788"
Nov 02 11:55:32.452606 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.452581    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal"]
Nov 02 11:55:32.452694 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.452649    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 11:55:32.453486 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.453469    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 11:55:32.453561 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.453500    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 11:55:32.453561 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.453520    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 11:55:32.455702 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.455688    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 11:55:32.455856 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.455841    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.455900 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.455872    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 11:55:32.456366 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.456350    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 11:55:32.456448 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.456350    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 11:55:32.456448 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.456413    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 11:55:32.456448 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.456425    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 11:55:32.456448 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.456379    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 11:55:32.456448 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.456446    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 11:55:32.458480 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.458463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.458553 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.458490    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 11:55:32.459655 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.459641    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 11:55:32.460608 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.459669    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 11:55:32.460608 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.459685    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 11:55:32.466992 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.466958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe93821f99a8c26257f227cd244c0afa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"fe93821f99a8c26257f227cd244c0afa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.467087 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.466996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe93821f99a8c26257f227cd244c0afa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"fe93821f99a8c26257f227cd244c0afa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.467146 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.467091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1797323b44d1a789ea358554c2680a4d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"1797323b44d1a789ea358554c2680a4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.486547 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:32.486530    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-239.us-west-1.compute.internal\" not found" node="ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.489159 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:32.489121    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-239.us-west-1.compute.internal\" not found" node="ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.505417 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.505397    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 11:55:32.553643 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.553623    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 11:55:32.555168 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.555152    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 11:55:32.555923 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.555908    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.555964 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:32.555932    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-239.us-west-1.compute.internal\": node \"ip-10-0-133-239.us-west-1.compute.internal\" not found"
Nov 02 11:55:32.556490 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.556477    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 11:55:32.567402 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.567389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe93821f99a8c26257f227cd244c0afa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"fe93821f99a8c26257f227cd244c0afa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.567449 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.567410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe93821f99a8c26257f227cd244c0afa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"fe93821f99a8c26257f227cd244c0afa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.567449 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.567426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1797323b44d1a789ea358554c2680a4d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"1797323b44d1a789ea358554c2680a4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.567548 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.567470    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1797323b44d1a789ea358554c2680a4d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"1797323b44d1a789ea358554c2680a4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.567548 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.567485    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe93821f99a8c26257f227cd244c0afa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"fe93821f99a8c26257f227cd244c0afa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.567548 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.567523    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe93821f99a8c26257f227cd244c0afa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"fe93821f99a8c26257f227cd244c0afa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.664121 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.664067    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.725906 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.725881    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 11:55:32.726018 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.725989    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.783468 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.783451    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 11:55:32.789664 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.789650    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:32.791802 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:32.791789    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:55:33.099280 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.099210    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 11:55:33.099786 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.099313    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 11:55:33.099786 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.099347    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 11:55:33.099786 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.099377    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 11:55:33.099786 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:33.099376    2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aadda5a39e08548bea3cd7c71322c295-74c64663e2d0aa4a.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-239.us-west-1.compute.internal.18e4385efa816483\": read tcp 10.0.133.239:42608->77.112.245.159:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-239.us-west-1.compute.internal.18e4385efa816483  default   4888 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-239.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-239.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-239.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 11:55:32 +0000 UTC,LastTimestamp:2026-11-02 11:55:32.335076166 +0000 UTC m=+0.555280914,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-239.us-west-1.compute.internal,}"
Nov 02 11:55:33.138103 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.138085    2577 apiserver.go:52] "Watching apiserver"
Nov 02 11:55:33.162652 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.162635    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 11:55:33.394095 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.394031    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 11:50:32 +0000 UTC" deadline="2028-06-21 23:48:39.256109403 +0000 UTC"
Nov 02 11:55:33.394095 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.394055    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14339h53m5.862057084s"
Nov 02 11:55:33.397800 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.397783    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 11:55:33.398125 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.398106    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-wdmr4","openshift-dns/node-resolver-b4wmw","openshift-image-registry/node-ca-ddszj","openshift-multus/network-metrics-daemon-bqwff","kube-system/global-pull-secret-syncer-sw29m","kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-4gjgd","openshift-network-diagnostics/network-check-target-x4hmk","openshift-network-operator/iptables-alerter-5zprr","openshift-ovn-kubernetes/ovnkube-node-w62gt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal","openshift-multus/multus-7wp5n","kube-system/konnectivity-agent-k6kl8"]
Nov 02 11:55:33.400328 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.400315    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.402162 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.402145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:55:33.404065 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.404046    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:55:33.406037 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.406025    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:33.406092 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:33.406076    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqwff" podUID="cbfec7e2-d13a-483b-8583-5f19481690c1"
Nov 02 11:55:33.407879 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.407868    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:33.407921 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:33.407912    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw29m" podUID="8161b19f-1a15-4f7d-9232-a9248995abf6"
Nov 02 11:55:33.409777 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.409766    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.411691 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.411674    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:33.411750 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:33.411721    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4hmk" podUID="2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab"
Nov 02 11:55:33.413801 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.413785    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:55:33.415872 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.415858    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.417825 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.417811    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.420197 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.420174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.422161 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.422145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k6kl8"
Nov 02 11:55:33.449201 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.449187    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 11:55:33.449316 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.449303    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 11:55:33.449367 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.449314    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gpdd6\""
Nov 02 11:55:33.451509 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.451493    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 11:55:33.451847 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.451830    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 11:55:33.451891 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.451872    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q864t\""
Nov 02 11:55:33.452705 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.452690    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 11:55:33.452764 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.452733    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 11:55:33.452809 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.452770    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 11:55:33.452857 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.452824    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4pvvq\""
Nov 02 11:55:33.465823 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.465809    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 11:55:33.472867 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.472851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-lib-modules\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.472942 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.472873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-cni-netd\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.472942 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.472887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-sys-fs\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.472942 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.472904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-cni-bin\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.473069 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.472944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/569eaf0a-207c-4686-ab02-82e91b45d285-host\") pod \"node-ca-ddszj\" (UID: \"569eaf0a-207c-4686-ab02-82e91b45d285\") " pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:55:33.473069 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.472964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-system-cni-dir\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.473069 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-kubernetes\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.473069 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysctl-d\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.473175 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-var-lib-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.473175 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-device-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.473175 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473113    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:33.473296 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-etc-kubernetes\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.473296 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.473296 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473225    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmw6l\" (UniqueName: \"kubernetes.io/projected/3199abe8-4925-4fba-a291-860f9b7394b9-kube-api-access-pmw6l\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.473296 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473246    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-host-slash\") pod \"iptables-alerter-5zprr\" (UID: \"751f0932-c6b4-4ee6-a4bf-f096e7d27e62\") " pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:55:33.473458 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473323    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxhnp\" (UniqueName: \"kubernetes.io/projected/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-kube-api-access-xxhnp\") pod \"iptables-alerter-5zprr\" (UID: \"751f0932-c6b4-4ee6-a4bf-f096e7d27e62\") " pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:55:33.473458 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473357    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-run-ovn-kubernetes\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.473458 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-cni-bin\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.473458 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-socket-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.473458 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-systemd-units\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.473616 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-log-socket\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.473616 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-conf-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.473616 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-multus-certs\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.473616 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-modprobe-d\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.473616 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473576    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-ovn\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.473616 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473596    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-registration-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.473866 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473616    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhsj8\" (UniqueName: \"kubernetes.io/projected/5592809c-08dd-46c1-be01-d1da5d6d54b7-kube-api-access-hhsj8\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.473866 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-hostroot\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.473866 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.473866 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b7058574-e988-4cc5-8a0c-fa961b9080c1-agent-certs\") pod \"konnectivity-agent-k6kl8\" (UID: \"b7058574-e988-4cc5-8a0c-fa961b9080c1\") " pod="kube-system/konnectivity-agent-k6kl8"
Nov 02 11:55:33.473866 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473717    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lftk\" (UniqueName: \"kubernetes.io/projected/cbfec7e2-d13a-483b-8583-5f19481690c1-kube-api-access-5lftk\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:33.473866 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-systemd\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.473866 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-node-log\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.473866 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovn-node-metrics-cert\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.473866 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysctl-conf\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.474279 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-sys\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.474279 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3199abe8-4925-4fba-a291-860f9b7394b9-etc-tuned\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.474279 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:33.474279 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-kubelet\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.474279 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.473952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdv8n\" (UniqueName: \"kubernetes.io/projected/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-kube-api-access-xdv8n\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.474279 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/569eaf0a-207c-4686-ab02-82e91b45d285-serviceca\") pod \"node-ca-ddszj\" (UID: \"569eaf0a-207c-4686-ab02-82e91b45d285\") " pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:55:33.474279 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.474279 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-socket-dir-parent\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.474279 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.474279 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474109    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-k8s-cni-cncf-io\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.474279 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t47mb\" (UniqueName: \"kubernetes.io/projected/836a65d8-536a-4183-9f71-4761970d5d15-kube-api-access-t47mb\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.474279 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8161b19f-1a15-4f7d-9232-a9248995abf6-kubelet-config\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:33.474279 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cni-binary-copy\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.474279 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474182    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysconfig\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.474279 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-var-lib-kubelet\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.474279 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-host\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.474754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-run-netns\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.474754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-cni-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.474754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-netns\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.474754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-kubelet\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.474754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/836a65d8-536a-4183-9f71-4761970d5d15-multus-daemon-config\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.474754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snxcb\" (UniqueName: \"kubernetes.io/projected/569eaf0a-207c-4686-ab02-82e91b45d285-kube-api-access-snxcb\") pod \"node-ca-ddszj\" (UID: \"569eaf0a-207c-4686-ab02-82e91b45d285\") " pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:55:33.474754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-systemd\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.474754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3199abe8-4925-4fba-a291-860f9b7394b9-tmp\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.474754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.474754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474421    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovnkube-script-lib\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.474754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-cni-multus\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.474754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474473    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f7641b1-828f-46e8-b276-293aa6ab5615-hosts-file\") pod \"node-resolver-b4wmw\" (UID: \"0f7641b1-828f-46e8-b276-293aa6ab5615\") " pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:55:33.474754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cnibin\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.474754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-run\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.474754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474541    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovnkube-config\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.474754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-system-cni-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.474754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/836a65d8-536a-4183-9f71-4761970d5d15-cni-binary-copy\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.475223 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjlf7\" (UniqueName: \"kubernetes.io/projected/0f7641b1-828f-46e8-b276-293aa6ab5615-kube-api-access-rjlf7\") pod \"node-resolver-b4wmw\" (UID: \"0f7641b1-828f-46e8-b276-293aa6ab5615\") " pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:55:33.475223 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.475223 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-iptables-alerter-script\") pod \"iptables-alerter-5zprr\" (UID: \"751f0932-c6b4-4ee6-a4bf-f096e7d27e62\") " pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:55:33.475223 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-slash\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.475223 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f7641b1-828f-46e8-b276-293aa6ab5615-tmp-dir\") pod \"node-resolver-b4wmw\" (UID: \"0f7641b1-828f-46e8-b276-293aa6ab5615\") " pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:55:33.475223 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474754    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8161b19f-1a15-4f7d-9232-a9248995abf6-dbus\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:33.475223 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbbm7\" (UniqueName: \"kubernetes.io/projected/918aeea0-5975-4be5-a2f4-530e6c38f1ad-kube-api-access-tbbm7\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.475223 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b7058574-e988-4cc5-8a0c-fa961b9080c1-konnectivity-ca\") pod \"konnectivity-agent-k6kl8\" (UID: \"b7058574-e988-4cc5-8a0c-fa961b9080c1\") " pod="kube-system/konnectivity-agent-k6kl8"
Nov 02 11:55:33.475223 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-etc-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.475223 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.475223 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-env-overrides\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.475223 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-cnibin\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.475223 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.474931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srjx5\" (UniqueName: \"kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5\") pod \"network-check-target-x4hmk\" (UID: \"2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab\") " pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:33.542364 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542347    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 11:55:33.542364 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542360    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 11:55:33.542470 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542436    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 11:55:33.542513 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542487    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8f6rg"
Nov 02 11:55:33.542681 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542663    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 11:55:33.542754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542740    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 11:55:33.542806 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542793    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 11:55:33.542836 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542799    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 11:55:33.542836 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542804    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 11:55:33.542836 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542814    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 11:55:33.542836 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542813    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hx94g\""
Nov 02 11:55:33.543060 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542860    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 11:55:33.543060 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542873    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 11:55:33.543060 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542900    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 11:55:33.543060 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542906    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 11:55:33.543060 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542913    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 11:55:33.543060 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542902    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 11:55:33.543060 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542930    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 11:55:33.543060 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542940    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 11:55:33.543060 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.542951    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 11:55:33.543060 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.543049    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2rrmx\""
Nov 02 11:55:33.543060 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.543050    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 11:55:33.561247 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.561227    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mvmxd\""
Nov 02 11:55:33.561410 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.561396    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x7vnq\""
Nov 02 11:55:33.575745 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.575732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-iptables-alerter-script\") pod \"iptables-alerter-5zprr\" (UID: \"751f0932-c6b4-4ee6-a4bf-f096e7d27e62\") " pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:55:33.575812 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.575751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-slash\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.575812 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.575767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f7641b1-828f-46e8-b276-293aa6ab5615-tmp-dir\") pod \"node-resolver-b4wmw\" (UID: \"0f7641b1-828f-46e8-b276-293aa6ab5615\") " pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:55:33.575812 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.575780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8161b19f-1a15-4f7d-9232-a9248995abf6-dbus\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:33.575812 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.575794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbbm7\" (UniqueName: \"kubernetes.io/projected/918aeea0-5975-4be5-a2f4-530e6c38f1ad-kube-api-access-tbbm7\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.575812 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.575807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b7058574-e988-4cc5-8a0c-fa961b9080c1-konnectivity-ca\") pod \"konnectivity-agent-k6kl8\" (UID: \"b7058574-e988-4cc5-8a0c-fa961b9080c1\") " pod="kube-system/konnectivity-agent-k6kl8"
Nov 02 11:55:33.576052 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.575826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-etc-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.576052 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.575846    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-slash\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.576052 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.575847    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.576052 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.575887    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.576052 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.575893    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-etc-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.576052 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.575900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-env-overrides\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.576052 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.575938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-cnibin\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.576052 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.575983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srjx5\" (UniqueName: \"kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5\") pod \"network-check-target-x4hmk\" (UID: \"2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab\") " pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:33.576052 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576001    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8161b19f-1a15-4f7d-9232-a9248995abf6-dbus\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:33.576052 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-lib-modules\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.576052 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-cni-netd\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-sys-fs\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-cni-bin\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576094    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f7641b1-828f-46e8-b276-293aa6ab5615-tmp-dir\") pod \"node-resolver-b4wmw\" (UID: \"0f7641b1-828f-46e8-b276-293aa6ab5615\") " pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/569eaf0a-207c-4686-ab02-82e91b45d285-host\") pod \"node-ca-ddszj\" (UID: \"569eaf0a-207c-4686-ab02-82e91b45d285\") " pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-system-cni-dir\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-kubernetes\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysctl-d\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576183    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-cnibin\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-var-lib-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-device-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576233    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-kubernetes\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576224    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/569eaf0a-207c-4686-ab02-82e91b45d285-host\") pod \"node-ca-ddszj\" (UID: \"569eaf0a-207c-4686-ab02-82e91b45d285\") " pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576274    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-cni-netd\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-etc-kubernetes\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:33.576319    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 11:55:33.576531 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576329    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-sys-fs\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.577256 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmw6l\" (UniqueName: \"kubernetes.io/projected/3199abe8-4925-4fba-a291-860f9b7394b9-kube-api-access-pmw6l\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.577256 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576174    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-lib-modules\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.577256 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:33.576371    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret podName:8161b19f-1a15-4f7d-9232-a9248995abf6 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:34.076354425 +0000 UTC m=+2.296559159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret") pod "global-pull-secret-syncer-sw29m" (UID: "8161b19f-1a15-4f7d-9232-a9248995abf6") : object "kube-system"/"original-pull-secret" not registered
Nov 02 11:55:33.577256 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576373    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-env-overrides\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.577256 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576380    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-etc-kubernetes\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.577256 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576389    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-iptables-alerter-script\") pod \"iptables-alerter-5zprr\" (UID: \"751f0932-c6b4-4ee6-a4bf-f096e7d27e62\") " pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:55:33.577256 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576430    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-device-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.577256 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576464    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-var-lib-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.577256 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-host-slash\") pod \"iptables-alerter-5zprr\" (UID: \"751f0932-c6b4-4ee6-a4bf-f096e7d27e62\") " pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:55:33.577256 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576499    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysctl-d\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.577256 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576501    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.577256 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhnp\" (UniqueName: \"kubernetes.io/projected/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-kube-api-access-xxhnp\") pod \"iptables-alerter-5zprr\" (UID: \"751f0932-c6b4-4ee6-a4bf-f096e7d27e62\") " pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:55:33.577256 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576535    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-system-cni-dir\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.577256 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-run-ovn-kubernetes\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.577256 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-cni-bin\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.577256 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576581    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-host-slash\") pod \"iptables-alerter-5zprr\" (UID: \"751f0932-c6b4-4ee6-a4bf-f096e7d27e62\") " pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576590    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-cni-bin\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-socket-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-systemd-units\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576634    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-run-ovn-kubernetes\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-log-socket\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576668    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-conf-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576674    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-systemd-units\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576637    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-cni-bin\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-multus-certs\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576716    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-socket-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576725    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-log-socket\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-modprobe-d\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576741    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-conf-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576759    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-multus-certs\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-ovn\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576786    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-ovn\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-registration-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.577919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576798    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-modprobe-d\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.578650 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhsj8\" (UniqueName: \"kubernetes.io/projected/5592809c-08dd-46c1-be01-d1da5d6d54b7-kube-api-access-hhsj8\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.578650 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576828    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-registration-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.578650 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-hostroot\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.578650 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.578650 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576882    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-hostroot\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.578650 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b7058574-e988-4cc5-8a0c-fa961b9080c1-agent-certs\") pod \"konnectivity-agent-k6kl8\" (UID: \"b7058574-e988-4cc5-8a0c-fa961b9080c1\") " pod="kube-system/konnectivity-agent-k6kl8"
Nov 02 11:55:33.578650 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lftk\" (UniqueName: \"kubernetes.io/projected/cbfec7e2-d13a-483b-8583-5f19481690c1-kube-api-access-5lftk\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:33.578650 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-systemd\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.578650 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.576998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-node-log\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.578650 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovn-node-metrics-cert\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.578650 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577030    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-systemd\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.578650 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysctl-conf\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.578650 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-sys\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.578650 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577077    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-node-log\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.578650 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3199abe8-4925-4fba-a291-860f9b7394b9-etc-tuned\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.578650 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:33.578650 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-kubelet\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.579283 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdv8n\" (UniqueName: \"kubernetes.io/projected/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-kube-api-access-xdv8n\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.579283 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577163    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-sys\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.579283 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577172    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/569eaf0a-207c-4686-ab02-82e91b45d285-serviceca\") pod \"node-ca-ddszj\" (UID: \"569eaf0a-207c-4686-ab02-82e91b45d285\") " pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:55:33.579283 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.579283 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577205    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysctl-conf\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.579283 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-socket-dir-parent\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.579283 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577259    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b7058574-e988-4cc5-8a0c-fa961b9080c1-konnectivity-ca\") pod \"konnectivity-agent-k6kl8\" (UID: \"b7058574-e988-4cc5-8a0c-fa961b9080c1\") " pod="kube-system/konnectivity-agent-k6kl8"
Nov 02 11:55:33.579283 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577266    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-socket-dir-parent\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.579283 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577307    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-kubelet\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.579283 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:33.577309    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 11:55:33.579283 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577317    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.579283 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577317    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 11:55:33.579283 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.579283 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:33.577357    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs podName:cbfec7e2-d13a-483b-8583-5f19481690c1 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:34.077343679 +0000 UTC m=+2.297548424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs") pod "network-metrics-daemon-bqwff" (UID: "cbfec7e2-d13a-483b-8583-5f19481690c1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 11:55:33.579283 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-k8s-cni-cncf-io\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.579283 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577383    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.579283 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577391    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577399    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t47mb\" (UniqueName: \"kubernetes.io/projected/836a65d8-536a-4183-9f71-4761970d5d15-kube-api-access-t47mb\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8161b19f-1a15-4f7d-9232-a9248995abf6-kubelet-config\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577468    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8161b19f-1a15-4f7d-9232-a9248995abf6-kubelet-config\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577473    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-k8s-cni-cncf-io\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cni-binary-copy\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysconfig\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-var-lib-kubelet\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-host\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577569    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysconfig\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577589    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/569eaf0a-207c-4686-ab02-82e91b45d285-serviceca\") pod \"node-ca-ddszj\" (UID: \"569eaf0a-207c-4686-ab02-82e91b45d285\") " pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577615    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-host\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577616    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-var-lib-kubelet\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-run-netns\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-cni-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577692    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-run-netns\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-netns\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577724    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-netns\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.579751 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577741    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-kubelet\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577760    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/836a65d8-536a-4183-9f71-4761970d5d15-multus-daemon-config\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577768    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-kubelet\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577774    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snxcb\" (UniqueName: \"kubernetes.io/projected/569eaf0a-207c-4686-ab02-82e91b45d285-kube-api-access-snxcb\") pod \"node-ca-ddszj\" (UID: \"569eaf0a-207c-4686-ab02-82e91b45d285\") " pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577743    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-cni-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-systemd\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3199abe8-4925-4fba-a291-860f9b7394b9-tmp\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577830    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-systemd\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovnkube-script-lib\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577877    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cni-binary-copy\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-cni-multus\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f7641b1-828f-46e8-b276-293aa6ab5615-hosts-file\") pod \"node-resolver-b4wmw\" (UID: \"0f7641b1-828f-46e8-b276-293aa6ab5615\") " pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577931    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cnibin\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577988    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-cni-multus\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577995    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cnibin\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.577992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-run\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.580300 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.578028    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-run\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.580790 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.578047    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f7641b1-828f-46e8-b276-293aa6ab5615-hosts-file\") pod \"node-resolver-b4wmw\" (UID: \"0f7641b1-828f-46e8-b276-293aa6ab5615\") " pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:55:33.580790 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.578065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovnkube-config\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.580790 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.578086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-system-cni-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.580790 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.578106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/836a65d8-536a-4183-9f71-4761970d5d15-cni-binary-copy\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.580790 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.578126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjlf7\" (UniqueName: \"kubernetes.io/projected/0f7641b1-828f-46e8-b276-293aa6ab5615-kube-api-access-rjlf7\") pod \"node-resolver-b4wmw\" (UID: \"0f7641b1-828f-46e8-b276-293aa6ab5615\") " pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:55:33.580790 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.578146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.580790 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.578160    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-system-cni-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.580790 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.578363    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/836a65d8-536a-4183-9f71-4761970d5d15-multus-daemon-config\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.580790 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.578527    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovnkube-config\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.580790 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.578634    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/836a65d8-536a-4183-9f71-4761970d5d15-cni-binary-copy\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.580790 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.579168    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.580790 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.580365    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3199abe8-4925-4fba-a291-860f9b7394b9-etc-tuned\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.581138 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.581130    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3199abe8-4925-4fba-a291-860f9b7394b9-tmp\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.581346 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.581220    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovn-node-metrics-cert\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.589191 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.589170    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 11:55:33.589258 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.589215    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-g4drg\""
Nov 02 11:55:33.589258 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.589226    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 11:55:33.589258 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.589239    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bjwgn\""
Nov 02 11:55:33.598943 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.598926    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovnkube-script-lib\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.599059 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.599047    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b7058574-e988-4cc5-8a0c-fa961b9080c1-agent-certs\") pod \"konnectivity-agent-k6kl8\" (UID: \"b7058574-e988-4cc5-8a0c-fa961b9080c1\") " pod="kube-system/konnectivity-agent-k6kl8"
Nov 02 11:55:33.633461 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:33.633439    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 11:55:33.633461 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:33.633459    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 11:55:33.633577 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:33.633470    2577 projected.go:196] Error preparing data for projected volume kube-api-access-srjx5 for pod openshift-network-diagnostics/network-check-target-x4hmk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 11:55:33.633577 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:33.633531    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5 podName:2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab nodeName:}" failed. No retries permitted until 2026-11-02 11:55:34.133515993 +0000 UTC m=+2.353720737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-srjx5" (UniqueName: "kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5") pod "network-check-target-x4hmk" (UID: "2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 11:55:33.635566 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.635549    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhsj8\" (UniqueName: \"kubernetes.io/projected/5592809c-08dd-46c1-be01-d1da5d6d54b7-kube-api-access-hhsj8\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.635806 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.635793    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmw6l\" (UniqueName: \"kubernetes.io/projected/3199abe8-4925-4fba-a291-860f9b7394b9-kube-api-access-pmw6l\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.638455 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.638432    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxhnp\" (UniqueName: \"kubernetes.io/projected/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-kube-api-access-xxhnp\") pod \"iptables-alerter-5zprr\" (UID: \"751f0932-c6b4-4ee6-a4bf-f096e7d27e62\") " pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:55:33.639236 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.639212    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdv8n\" (UniqueName: \"kubernetes.io/projected/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-kube-api-access-xdv8n\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.639361 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.639337    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjlf7\" (UniqueName: \"kubernetes.io/projected/0f7641b1-828f-46e8-b276-293aa6ab5615-kube-api-access-rjlf7\") pod \"node-resolver-b4wmw\" (UID: \"0f7641b1-828f-46e8-b276-293aa6ab5615\") " pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:55:33.639460 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.639362    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snxcb\" (UniqueName: \"kubernetes.io/projected/569eaf0a-207c-4686-ab02-82e91b45d285-kube-api-access-snxcb\") pod \"node-ca-ddszj\" (UID: \"569eaf0a-207c-4686-ab02-82e91b45d285\") " pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:55:33.640286 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.640271    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbbm7\" (UniqueName: \"kubernetes.io/projected/918aeea0-5975-4be5-a2f4-530e6c38f1ad-kube-api-access-tbbm7\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.640506 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.640486    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lftk\" (UniqueName: \"kubernetes.io/projected/cbfec7e2-d13a-483b-8583-5f19481690c1-kube-api-access-5lftk\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:33.640734 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.640722    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t47mb\" (UniqueName: \"kubernetes.io/projected/836a65d8-536a-4183-9f71-4761970d5d15-kube-api-access-t47mb\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.720445 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.720391    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:55:33.737033 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.736958    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:55:33.747836 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.747813    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:55:33.776514 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.776491    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:55:33.777534 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.777517    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:55:33.806957 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.806941    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:33.819476 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.819462    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:55:33.832708 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.832685    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7wp5n"
Nov 02 11:55:33.837541 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.837525    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k6kl8"
Nov 02 11:55:33.846237 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:33.846204    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5592809c_08dd_46c1_be01_d1da5d6d54b7.slice/crio-a403d5c8339a18277bf2a6300d7fcd648ac158dd99dda437c4f8d268ef1ccff0 WatchSource:0}: Error finding container a403d5c8339a18277bf2a6300d7fcd648ac158dd99dda437c4f8d268ef1ccff0: Status 404 returned error can't find the container with id a403d5c8339a18277bf2a6300d7fcd648ac158dd99dda437c4f8d268ef1ccff0
Nov 02 11:55:33.846522 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:33.846503    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836a65d8_536a_4183_9f71_4761970d5d15.slice/crio-1284073a42cc36538dedb298665dddbb04d7b232c53be6480a5799a353fa4a85 WatchSource:0}: Error finding container 1284073a42cc36538dedb298665dddbb04d7b232c53be6480a5799a353fa4a85: Status 404 returned error can't find the container with id 1284073a42cc36538dedb298665dddbb04d7b232c53be6480a5799a353fa4a85
Nov 02 11:55:33.847452 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:33.847431    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe93821f99a8c26257f227cd244c0afa.slice/crio-b5632eeb6f484ba08f8999719e59c240ba6312687fdf7faa7115e8852adec820 WatchSource:0}: Error finding container b5632eeb6f484ba08f8999719e59c240ba6312687fdf7faa7115e8852adec820: Status 404 returned error can't find the container with id b5632eeb6f484ba08f8999719e59c240ba6312687fdf7faa7115e8852adec820
Nov 02 11:55:33.848485 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:33.848465    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1797323b44d1a789ea358554c2680a4d.slice/crio-1710e2a33d16ed503ec68bc7bec0797fdabee0474f2ccf93501753b3af0ec00f WatchSource:0}: Error finding container 1710e2a33d16ed503ec68bc7bec0797fdabee0474f2ccf93501753b3af0ec00f: Status 404 returned error can't find the container with id 1710e2a33d16ed503ec68bc7bec0797fdabee0474f2ccf93501753b3af0ec00f
Nov 02 11:55:33.849334 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:33.849316    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7058574_e988_4cc5_8a0c_fa961b9080c1.slice/crio-58d7f8633a39d4d6aeb31881cd3fa5d9fac6663d0f08770b25f58d5da4c7833d WatchSource:0}: Error finding container 58d7f8633a39d4d6aeb31881cd3fa5d9fac6663d0f08770b25f58d5da4c7833d: Status 404 returned error can't find the container with id 58d7f8633a39d4d6aeb31881cd3fa5d9fac6663d0f08770b25f58d5da4c7833d
Nov 02 11:55:33.851917 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:33.851908    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 11:55:34.081128 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:34.081057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:34.081128 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:34.081097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:34.081311 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:34.081186    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 11:55:34.081311 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:34.081196    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 11:55:34.081311 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:34.081236    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs podName:cbfec7e2-d13a-483b-8583-5f19481690c1 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:35.081220374 +0000 UTC m=+3.301425108 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs") pod "network-metrics-daemon-bqwff" (UID: "cbfec7e2-d13a-483b-8583-5f19481690c1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 11:55:34.081311 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:34.081248    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret podName:8161b19f-1a15-4f7d-9232-a9248995abf6 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:35.081242611 +0000 UTC m=+3.301447344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret") pod "global-pull-secret-syncer-sw29m" (UID: "8161b19f-1a15-4f7d-9232-a9248995abf6") : object "kube-system"/"original-pull-secret" not registered
Nov 02 11:55:34.145314 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:34.145291    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 11:55:34.181309 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:34.181281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srjx5\" (UniqueName: \"kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5\") pod \"network-check-target-x4hmk\" (UID: \"2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab\") " pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:34.181396 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:34.181386    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 11:55:34.181425 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:34.181397    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 11:55:34.181425 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:34.181405    2577 projected.go:196] Error preparing data for projected volume kube-api-access-srjx5 for pod openshift-network-diagnostics/network-check-target-x4hmk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 11:55:34.181481 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:34.181443    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5 podName:2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab nodeName:}" failed. No retries permitted until 2026-11-02 11:55:35.181431887 +0000 UTC m=+3.401636623 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-srjx5" (UniqueName: "kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5") pod "network-check-target-x4hmk" (UID: "2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 11:55:34.278019 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:34.278000    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 11:55:34.308875 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:34.308854    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 11:55:34.356393 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:34.356322    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal" event={"ID":"fe93821f99a8c26257f227cd244c0afa","Type":"ContainerStarted","Data":"b5632eeb6f484ba08f8999719e59c240ba6312687fdf7faa7115e8852adec820"}
Nov 02 11:55:34.357258 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:34.357239    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k6kl8" event={"ID":"b7058574-e988-4cc5-8a0c-fa961b9080c1","Type":"ContainerStarted","Data":"58d7f8633a39d4d6aeb31881cd3fa5d9fac6663d0f08770b25f58d5da4c7833d"}
Nov 02 11:55:34.358120 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:34.358104    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7wp5n" event={"ID":"836a65d8-536a-4183-9f71-4761970d5d15","Type":"ContainerStarted","Data":"1284073a42cc36538dedb298665dddbb04d7b232c53be6480a5799a353fa4a85"}
Nov 02 11:55:34.359080 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:34.359062    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal" event={"ID":"1797323b44d1a789ea358554c2680a4d","Type":"ContainerStarted","Data":"1710e2a33d16ed503ec68bc7bec0797fdabee0474f2ccf93501753b3af0ec00f"}
Nov 02 11:55:34.359963 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:34.359938    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7" event={"ID":"5592809c-08dd-46c1-be01-d1da5d6d54b7","Type":"ContainerStarted","Data":"a403d5c8339a18277bf2a6300d7fcd648ac158dd99dda437c4f8d268ef1ccff0"}
Nov 02 11:55:34.462951 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:34.462931    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7641b1_828f_46e8_b276_293aa6ab5615.slice/crio-a26c9da442602e856b17686f6f9cc1907cecace0a33e514730133db50daf891a WatchSource:0}: Error finding container a26c9da442602e856b17686f6f9cc1907cecace0a33e514730133db50daf891a: Status 404 returned error can't find the container with id a26c9da442602e856b17686f6f9cc1907cecace0a33e514730133db50daf891a
Nov 02 11:55:34.496120 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:34.496102    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918aeea0_5975_4be5_a2f4_530e6c38f1ad.slice/crio-93fc077760a4de14212716c28c0a334546ef6478f75595d7c732c792ff7945c4 WatchSource:0}: Error finding container 93fc077760a4de14212716c28c0a334546ef6478f75595d7c732c792ff7945c4: Status 404 returned error can't find the container with id 93fc077760a4de14212716c28c0a334546ef6478f75595d7c732c792ff7945c4
Nov 02 11:55:34.538451 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:34.538429    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751f0932_c6b4_4ee6_a4bf_f096e7d27e62.slice/crio-bb316f99149fcb3de0f5cc3836c5732d066048fce2ee449c85a18fd411d1b1a6 WatchSource:0}: Error finding container bb316f99149fcb3de0f5cc3836c5732d066048fce2ee449c85a18fd411d1b1a6: Status 404 returned error can't find the container with id bb316f99149fcb3de0f5cc3836c5732d066048fce2ee449c85a18fd411d1b1a6
Nov 02 11:55:34.539143 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:34.539124    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569eaf0a_207c_4686_ab02_82e91b45d285.slice/crio-b1e788a4fcc68154bacbfccaeaa4b363bab86ec9e0c805d63d666ac642e03106 WatchSource:0}: Error finding container b1e788a4fcc68154bacbfccaeaa4b363bab86ec9e0c805d63d666ac642e03106: Status 404 returned error can't find the container with id b1e788a4fcc68154bacbfccaeaa4b363bab86ec9e0c805d63d666ac642e03106
Nov 02 11:55:34.545276 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:34.545229    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 11:50:33 +0000 UTC" deadline="2028-06-15 03:39:11.413718554 +0000 UTC"
Nov 02 11:55:34.545276 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:34.545275    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14175h43m36.86844764s"
Nov 02 11:55:34.558929 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:34.558909    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d66d28c_8f94_4e4f_a07a_10b188c9bc6a.slice/crio-2588e069f31d8fed2b5d530cf6a7de8b128785c7699da05a74e4d57daa6c624a WatchSource:0}: Error finding container 2588e069f31d8fed2b5d530cf6a7de8b128785c7699da05a74e4d57daa6c624a: Status 404 returned error can't find the container with id 2588e069f31d8fed2b5d530cf6a7de8b128785c7699da05a74e4d57daa6c624a
Nov 02 11:55:34.575018 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:55:34.575001    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3199abe8_4925_4fba_a291_860f9b7394b9.slice/crio-83fa755d1951fb98d991e17c58f046477245b0df6f0527424d6b81f7ba6752d6 WatchSource:0}: Error finding container 83fa755d1951fb98d991e17c58f046477245b0df6f0527424d6b81f7ba6752d6: Status 404 returned error can't find the container with id 83fa755d1951fb98d991e17c58f046477245b0df6f0527424d6b81f7ba6752d6
Nov 02 11:55:35.088606 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:35.088571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:35.088766 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:35.088631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:35.088819 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:35.088767    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 11:55:35.088865 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:35.088827    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs podName:cbfec7e2-d13a-483b-8583-5f19481690c1 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:37.088809924 +0000 UTC m=+5.309014668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs") pod "network-metrics-daemon-bqwff" (UID: "cbfec7e2-d13a-483b-8583-5f19481690c1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 11:55:35.088935 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:35.088907    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 11:55:35.089006 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:35.088952    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret podName:8161b19f-1a15-4f7d-9232-a9248995abf6 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:37.088940434 +0000 UTC m=+5.309145179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret") pod "global-pull-secret-syncer-sw29m" (UID: "8161b19f-1a15-4f7d-9232-a9248995abf6") : object "kube-system"/"original-pull-secret" not registered
Nov 02 11:55:35.189367 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:35.189331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srjx5\" (UniqueName: \"kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5\") pod \"network-check-target-x4hmk\" (UID: \"2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab\") " pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:35.189828 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:35.189498    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 11:55:35.189828 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:35.189521    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 11:55:35.189828 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:35.189534    2577 projected.go:196] Error preparing data for projected volume kube-api-access-srjx5 for pod openshift-network-diagnostics/network-check-target-x4hmk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 11:55:35.189828 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:35.189586    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5 podName:2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab nodeName:}" failed. No retries permitted until 2026-11-02 11:55:37.189571908 +0000 UTC m=+5.409776655 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-srjx5" (UniqueName: "kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5") pod "network-check-target-x4hmk" (UID: "2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 11:55:35.353446 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:35.353344    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:35.353657 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:35.353478    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4hmk" podUID="2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab"
Nov 02 11:55:35.353657 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:35.353570    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:35.353844 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:35.353674    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqwff" podUID="cbfec7e2-d13a-483b-8583-5f19481690c1"
Nov 02 11:55:35.353844 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:35.353740    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:35.353844 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:35.353811    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw29m" podUID="8161b19f-1a15-4f7d-9232-a9248995abf6"
Nov 02 11:55:35.376573 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:35.376542    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wdmr4" event={"ID":"3199abe8-4925-4fba-a291-860f9b7394b9","Type":"ContainerStarted","Data":"83fa755d1951fb98d991e17c58f046477245b0df6f0527424d6b81f7ba6752d6"}
Nov 02 11:55:35.385036 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:35.385006    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gjgd" event={"ID":"918aeea0-5975-4be5-a2f4-530e6c38f1ad","Type":"ContainerStarted","Data":"93fc077760a4de14212716c28c0a334546ef6478f75595d7c732c792ff7945c4"}
Nov 02 11:55:35.387943 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:35.387910    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt" event={"ID":"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a","Type":"ContainerStarted","Data":"2588e069f31d8fed2b5d530cf6a7de8b128785c7699da05a74e4d57daa6c624a"}
Nov 02 11:55:35.392877 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:35.392847    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ddszj" event={"ID":"569eaf0a-207c-4686-ab02-82e91b45d285","Type":"ContainerStarted","Data":"b1e788a4fcc68154bacbfccaeaa4b363bab86ec9e0c805d63d666ac642e03106"}
Nov 02 11:55:35.396709 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:35.396686    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5zprr" event={"ID":"751f0932-c6b4-4ee6-a4bf-f096e7d27e62","Type":"ContainerStarted","Data":"bb316f99149fcb3de0f5cc3836c5732d066048fce2ee449c85a18fd411d1b1a6"}
Nov 02 11:55:35.398370 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:35.398347    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b4wmw" event={"ID":"0f7641b1-828f-46e8-b276-293aa6ab5615","Type":"ContainerStarted","Data":"a26c9da442602e856b17686f6f9cc1907cecace0a33e514730133db50daf891a"}
Nov 02 11:55:35.546466 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:35.546389    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 11:50:33 +0000 UTC" deadline="2028-04-16 13:49:37.201512206 +0000 UTC"
Nov 02 11:55:35.546466 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:35.546447    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12745h54m1.655069613s"
Nov 02 11:55:37.105240 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:37.105178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:37.105240 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:37.105236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:37.105770 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:37.105376    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 11:55:37.105770 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:37.105437    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs podName:cbfec7e2-d13a-483b-8583-5f19481690c1 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:41.105420595 +0000 UTC m=+9.325625349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs") pod "network-metrics-daemon-bqwff" (UID: "cbfec7e2-d13a-483b-8583-5f19481690c1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 11:55:37.105770 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:37.105519    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 11:55:37.105770 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:37.105555    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret podName:8161b19f-1a15-4f7d-9232-a9248995abf6 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:41.105543362 +0000 UTC m=+9.325748158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret") pod "global-pull-secret-syncer-sw29m" (UID: "8161b19f-1a15-4f7d-9232-a9248995abf6") : object "kube-system"/"original-pull-secret" not registered
Nov 02 11:55:37.206455 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:37.206398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srjx5\" (UniqueName: \"kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5\") pod \"network-check-target-x4hmk\" (UID: \"2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab\") " pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:37.206638 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:37.206608    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 11:55:37.206638 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:37.206628    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 11:55:37.206638 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:37.206638    2577 projected.go:196] Error preparing data for projected volume kube-api-access-srjx5 for pod openshift-network-diagnostics/network-check-target-x4hmk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 11:55:37.206789 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:37.206701    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5 podName:2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab nodeName:}" failed. No retries permitted until 2026-11-02 11:55:41.206685856 +0000 UTC m=+9.426890602 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-srjx5" (UniqueName: "kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5") pod "network-check-target-x4hmk" (UID: "2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 11:55:37.352493 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:37.352451    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:37.352685 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:37.352550    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4hmk" podUID="2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab"
Nov 02 11:55:37.352685 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:37.352596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:37.352685 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:37.352649    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqwff" podUID="cbfec7e2-d13a-483b-8583-5f19481690c1"
Nov 02 11:55:37.352685 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:37.352685    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:37.352860 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:37.352726    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw29m" podUID="8161b19f-1a15-4f7d-9232-a9248995abf6"
Nov 02 11:55:38.426089 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:38.426044    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k6kl8" event={"ID":"b7058574-e988-4cc5-8a0c-fa961b9080c1","Type":"ContainerStarted","Data":"cc7066d1433ebcf09cfd3a00d561975d33f2110b599b4c6d703e8c657d324ed8"}
Nov 02 11:55:38.430001 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:38.429769    2577 generic.go:356] "Generic (PLEG): container finished" podID="fe93821f99a8c26257f227cd244c0afa" containerID="63fe7b8c30079dd31ab6b81e61b14ebd354990a43eac84e2b217687acbf12bf2" exitCode=0
Nov 02 11:55:38.430001 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:38.429847    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal" event={"ID":"fe93821f99a8c26257f227cd244c0afa","Type":"ContainerDied","Data":"63fe7b8c30079dd31ab6b81e61b14ebd354990a43eac84e2b217687acbf12bf2"}
Nov 02 11:55:38.435598 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:38.435183    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal" event={"ID":"1797323b44d1a789ea358554c2680a4d","Type":"ContainerStarted","Data":"73874b9877ed0d0c950db122b3aff86d43fe7578d9cfe2851b5e96d8d7f22d2a"}
Nov 02 11:55:38.438994 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:38.438291    2577 generic.go:356] "Generic (PLEG): container finished" podID="918aeea0-5975-4be5-a2f4-530e6c38f1ad" containerID="9c0ae90acb31cee84c8d6a4fc25ccde6b48f747b2fd1115afa0aaecdec8dffdb" exitCode=0
Nov 02 11:55:38.438994 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:38.438365    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gjgd" event={"ID":"918aeea0-5975-4be5-a2f4-530e6c38f1ad","Type":"ContainerDied","Data":"9c0ae90acb31cee84c8d6a4fc25ccde6b48f747b2fd1115afa0aaecdec8dffdb"}
Nov 02 11:55:38.548671 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:38.548601    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k6kl8" podStartSLOduration=3.007796972 podStartE2EDuration="6.548583288s" totalImagesPullingTime="3.540786316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:38.54817774 +0000 UTC m=+6.768382527" watchObservedRunningTime="2026-11-02 11:55:38.548583288 +0000 UTC m=+6.768788044"
Nov 02 11:55:38.781852 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:38.781791    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal" podStartSLOduration=6.781773382 podStartE2EDuration="6.781773382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:38.780595276 +0000 UTC m=+7.000800032" watchObservedRunningTime="2026-11-02 11:55:38.781773382 +0000 UTC m=+7.001978139"
Nov 02 11:55:38.797370 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:38.797345    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k6kl8"
Nov 02 11:55:38.797854 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:38.797830    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k6kl8"
Nov 02 11:55:39.353399 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:39.353369    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:39.353547 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:39.353501    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqwff" podUID="cbfec7e2-d13a-483b-8583-5f19481690c1"
Nov 02 11:55:39.353884 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:39.353863    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:39.354004 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:39.353967    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw29m" podUID="8161b19f-1a15-4f7d-9232-a9248995abf6"
Nov 02 11:55:39.354080 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:39.354064    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:39.354191 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:39.354170    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4hmk" podUID="2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab"
Nov 02 11:55:39.446697 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:39.446665    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal" event={"ID":"fe93821f99a8c26257f227cd244c0afa","Type":"ContainerStarted","Data":"3495aff3a966668a9ac0523e0320d66ec8c82831844e46c61deffe02f10d7387"}
Nov 02 11:55:39.564926 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:39.564865    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal" podStartSLOduration=7.564844946 podStartE2EDuration="7.564844946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:39.56419958 +0000 UTC m=+7.784404337" watchObservedRunningTime="2026-11-02 11:55:39.564844946 +0000 UTC m=+7.785049703"
Nov 02 11:55:40.452896 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:40.452813    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7" event={"ID":"5592809c-08dd-46c1-be01-d1da5d6d54b7","Type":"ContainerStarted","Data":"e09d824d876e394a35a4b088122215753d4c77d5335422a789ce3fbf31e72f4a"}
Nov 02 11:55:40.457919 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:40.457893    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ddszj" event={"ID":"569eaf0a-207c-4686-ab02-82e91b45d285","Type":"ContainerStarted","Data":"bfd71a6d1e4d7eef5ee36053706736a5a328d8f968a7ca78e12bf9f4bae30b36"}
Nov 02 11:55:40.458065 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:40.458052    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 11:55:41.138839 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:41.138038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:41.138839 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:41.138100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:41.138839 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:41.138249    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 11:55:41.138839 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:41.138313    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs podName:cbfec7e2-d13a-483b-8583-5f19481690c1 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:49.138292596 +0000 UTC m=+17.358497366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs") pod "network-metrics-daemon-bqwff" (UID: "cbfec7e2-d13a-483b-8583-5f19481690c1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 11:55:41.138839 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:41.138702    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 11:55:41.138839 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:41.138754    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret podName:8161b19f-1a15-4f7d-9232-a9248995abf6 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:49.138739852 +0000 UTC m=+17.358944598 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret") pod "global-pull-secret-syncer-sw29m" (UID: "8161b19f-1a15-4f7d-9232-a9248995abf6") : object "kube-system"/"original-pull-secret" not registered
Nov 02 11:55:41.239101 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:41.238795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srjx5\" (UniqueName: \"kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5\") pod \"network-check-target-x4hmk\" (UID: \"2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab\") " pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:41.239101 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:41.238953    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 11:55:41.239101 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:41.238987    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 11:55:41.239101 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:41.239001    2577 projected.go:196] Error preparing data for projected volume kube-api-access-srjx5 for pod openshift-network-diagnostics/network-check-target-x4hmk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 11:55:41.239101 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:41.239055    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5 podName:2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab nodeName:}" failed. No retries permitted until 2026-11-02 11:55:49.239037706 +0000 UTC m=+17.459242441 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-srjx5" (UniqueName: "kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5") pod "network-check-target-x4hmk" (UID: "2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 11:55:41.353955 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:41.353486    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:41.353955 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:41.353527    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:41.353955 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:41.353488    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:41.353955 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:41.353626    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4hmk" podUID="2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab"
Nov 02 11:55:41.353955 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:41.353734    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqwff" podUID="cbfec7e2-d13a-483b-8583-5f19481690c1"
Nov 02 11:55:41.353955 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:41.353814    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw29m" podUID="8161b19f-1a15-4f7d-9232-a9248995abf6"
Nov 02 11:55:42.254664 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:42.254623    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 11:55:42.469696 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:42.469663    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7wp5n" event={"ID":"836a65d8-536a-4183-9f71-4761970d5d15","Type":"ContainerStarted","Data":"98ab155d90162e56248022eca138d62b9c9d4160431ed0fe41ffc2c9e3ce6eff"}
Nov 02 11:55:42.473716 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:42.473643    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7" event={"ID":"5592809c-08dd-46c1-be01-d1da5d6d54b7","Type":"ContainerStarted","Data":"20366bff496a28a9904453d81629594af2e8bacea34a5edffce88b20fd7dcb25"}
Nov 02 11:55:42.607017 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:42.605104    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ddszj" podStartSLOduration=5.23579413 podStartE2EDuration="10.605072403s" totalImagesPullingTime="5.369278273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:40.574895494 +0000 UTC m=+8.795100250" watchObservedRunningTime="2026-11-02 11:55:42.605072403 +0000 UTC m=+10.825277160"
Nov 02 11:55:42.607017 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:42.606056    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7wp5n" podStartSLOduration=2.352602976 podStartE2EDuration="10.606045857s" totalImagesPullingTime="8.253442881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:42.601753996 +0000 UTC m=+10.821958744" watchObservedRunningTime="2026-11-02 11:55:42.606045857 +0000 UTC m=+10.826250612"
Nov 02 11:55:43.239955 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:43.239590    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T11:55:42.254645514Z","UUID":"c8cae43e-ac92-4e34-b576-de74c4fb33b1","Handler":null,"Name":"","Endpoint":""}
Nov 02 11:55:43.241522 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:43.241284    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 11:55:43.241522 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:43.241313    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 11:55:43.353428 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:43.352897    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:43.353428 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:43.352920    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:43.353428 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:43.353036    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw29m" podUID="8161b19f-1a15-4f7d-9232-a9248995abf6"
Nov 02 11:55:43.353428 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:43.353231    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:43.353428 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:43.353329    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqwff" podUID="cbfec7e2-d13a-483b-8583-5f19481690c1"
Nov 02 11:55:43.353428 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:43.353395    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4hmk" podUID="2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab"
Nov 02 11:55:43.479145 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:43.479109    2577 generic.go:356] "Generic (PLEG): container finished" podID="918aeea0-5975-4be5-a2f4-530e6c38f1ad" containerID="3aa81d288a6de43b89a77559018a68f13a49785353de96bb5d271c462e87ed23" exitCode=0
Nov 02 11:55:43.479296 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:43.479217    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gjgd" event={"ID":"918aeea0-5975-4be5-a2f4-530e6c38f1ad","Type":"ContainerDied","Data":"3aa81d288a6de43b89a77559018a68f13a49785353de96bb5d271c462e87ed23"}
Nov 02 11:55:43.484557 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:43.484533    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b4wmw" event={"ID":"0f7641b1-828f-46e8-b276-293aa6ab5615","Type":"ContainerStarted","Data":"e82c8eaf5822adf39d3de65edc381d1bd478ed03afcb857d162cc61dd25ec2c0"}
Nov 02 11:55:44.487271 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:44.487236    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5zprr" event={"ID":"751f0932-c6b4-4ee6-a4bf-f096e7d27e62","Type":"ContainerStarted","Data":"6fc0cb1ca19d20bf47339924a2a67aa55d0f4238e148e87f4a3d96652eab00e5"}
Nov 02 11:55:44.489693 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:44.489665    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7" event={"ID":"5592809c-08dd-46c1-be01-d1da5d6d54b7","Type":"ContainerStarted","Data":"226199b247be4c0e0d2e080db09735126ca8484826bf295c4fb04ef22f1f99a4"}
Nov 02 11:55:44.603925 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:44.603873    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-b4wmw" podStartSLOduration=4.590879691 podStartE2EDuration="12.603854899s" totalImagesPullingTime="8.012975208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:43.721121552 +0000 UTC m=+11.941326309" watchObservedRunningTime="2026-11-02 11:55:44.603854899 +0000 UTC m=+12.824059654"
Nov 02 11:55:44.724152 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:44.724112    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5zprr" podStartSLOduration=4.60912426 podStartE2EDuration="12.724098852s" totalImagesPullingTime="8.114974592s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:44.604854888 +0000 UTC m=+12.825059676" watchObservedRunningTime="2026-11-02 11:55:44.724098852 +0000 UTC m=+12.944303608"
Nov 02 11:55:44.724343 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:44.724314    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7" podStartSLOduration=3.232202588 podStartE2EDuration="12.724303199s" totalImagesPullingTime="9.492100611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:44.723654515 +0000 UTC m=+12.943859293" watchObservedRunningTime="2026-11-02 11:55:44.724303199 +0000 UTC m=+12.944507959"
Nov 02 11:55:45.353316 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:45.353283    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:45.353495 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:45.353396    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:45.353495 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:45.353406    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw29m" podUID="8161b19f-1a15-4f7d-9232-a9248995abf6"
Nov 02 11:55:45.353606 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:45.353520    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqwff" podUID="cbfec7e2-d13a-483b-8583-5f19481690c1"
Nov 02 11:55:45.353606 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:45.353547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:45.353695 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:45.353606    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4hmk" podUID="2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab"
Nov 02 11:55:45.494104 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:45.494068    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wdmr4" event={"ID":"3199abe8-4925-4fba-a291-860f9b7394b9","Type":"ContainerStarted","Data":"f54c291de39672ab612e159ce9b39bcf0ca8c1b15c61a6bf70f465e6c223b14a"}
Nov 02 11:55:45.496449 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:45.496415    2577 generic.go:356] "Generic (PLEG): container finished" podID="918aeea0-5975-4be5-a2f4-530e6c38f1ad" containerID="869ad29dcf3e35a22b4ba4234d7f801e7bf374d48afaa20ccbf26a937f60e733" exitCode=0
Nov 02 11:55:45.496584 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:45.496516    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gjgd" event={"ID":"918aeea0-5975-4be5-a2f4-530e6c38f1ad","Type":"ContainerDied","Data":"869ad29dcf3e35a22b4ba4234d7f801e7bf374d48afaa20ccbf26a937f60e733"}
Nov 02 11:55:45.642153 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:45.642063    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wdmr4" podStartSLOduration=3.545115975 podStartE2EDuration="13.642049282s" totalImagesPullingTime="10.096933307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:45.641853957 +0000 UTC m=+13.862058714" watchObservedRunningTime="2026-11-02 11:55:45.642049282 +0000 UTC m=+13.862254038"
Nov 02 11:55:47.352818 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:47.352588    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:47.353323 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:47.352603    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:47.353323 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:47.352923    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4hmk" podUID="2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab"
Nov 02 11:55:47.353323 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:47.352614    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:47.353323 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:47.353024    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqwff" podUID="cbfec7e2-d13a-483b-8583-5f19481690c1"
Nov 02 11:55:47.353323 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:47.353098    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw29m" podUID="8161b19f-1a15-4f7d-9232-a9248995abf6"
Nov 02 11:55:47.503823 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:47.503790    2577 generic.go:356] "Generic (PLEG): container finished" podID="918aeea0-5975-4be5-a2f4-530e6c38f1ad" containerID="7494acb94f3bd30f567f9f8571991f2d72b4378813a09a542ebbf5de6d5b579a" exitCode=0
Nov 02 11:55:47.503969 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:47.503845    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gjgd" event={"ID":"918aeea0-5975-4be5-a2f4-530e6c38f1ad","Type":"ContainerDied","Data":"7494acb94f3bd30f567f9f8571991f2d72b4378813a09a542ebbf5de6d5b579a"}
Nov 02 11:55:49.196867 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:49.196842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:49.197200 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:49.197007    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 11:55:49.197200 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:49.197074    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs podName:cbfec7e2-d13a-483b-8583-5f19481690c1 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:05.197055019 +0000 UTC m=+33.417259774 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs") pod "network-metrics-daemon-bqwff" (UID: "cbfec7e2-d13a-483b-8583-5f19481690c1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 11:55:49.197200 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:49.197009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:49.197200 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:49.197099    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 11:55:49.197200 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:49.197157    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret podName:8161b19f-1a15-4f7d-9232-a9248995abf6 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:05.197142862 +0000 UTC m=+33.417347601 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret") pod "global-pull-secret-syncer-sw29m" (UID: "8161b19f-1a15-4f7d-9232-a9248995abf6") : object "kube-system"/"original-pull-secret" not registered
Nov 02 11:55:49.298283 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:49.298133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srjx5\" (UniqueName: \"kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5\") pod \"network-check-target-x4hmk\" (UID: \"2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab\") " pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:49.298368 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:49.298309    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 11:55:49.298368 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:49.298334    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 11:55:49.298368 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:49.298347    2577 projected.go:196] Error preparing data for projected volume kube-api-access-srjx5 for pod openshift-network-diagnostics/network-check-target-x4hmk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 11:55:49.298472 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:49.298398    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5 podName:2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab nodeName:}" failed. No retries permitted until 2026-11-02 11:56:05.298385111 +0000 UTC m=+33.518589845 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-srjx5" (UniqueName: "kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5") pod "network-check-target-x4hmk" (UID: "2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 11:55:49.352501 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:49.352476    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:49.352605 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:49.352497    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:49.352605 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:49.352485    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:49.352669 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:49.352608    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqwff" podUID="cbfec7e2-d13a-483b-8583-5f19481690c1"
Nov 02 11:55:49.352711 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:49.352668    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw29m" podUID="8161b19f-1a15-4f7d-9232-a9248995abf6"
Nov 02 11:55:49.352769 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:49.352746    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4hmk" podUID="2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab"
Nov 02 11:55:49.510383 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:49.510359    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 11:55:49.510754 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:49.510724    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d66d28c-8f94-4e4f-a07a-10b188c9bc6a" containerID="093f77f5f86f98f54a23fc84e74d6708a935539196949ccf3c4975bd721b24a6" exitCode=1
Nov 02 11:55:49.510873 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:49.510778    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt" event={"ID":"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a","Type":"ContainerStarted","Data":"ab9ac90545e6608c6b5030b4fc095679b4f3a0fa1fe89a2dfda32461ff4b54b4"}
Nov 02 11:55:49.510873 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:49.510834    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt" event={"ID":"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a","Type":"ContainerStarted","Data":"b2a3b1e40f5d7c31d290abe229f1224687c98e16fa8fbd2c9edc329895a82afe"}
Nov 02 11:55:49.510873 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:49.510866    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt" event={"ID":"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a","Type":"ContainerStarted","Data":"96cc08c55e1ffa75b3e40d4702e772447bff7c27ca0332e9fae11cc16fd7ed65"}
Nov 02 11:55:49.510967 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:49.510897    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt" event={"ID":"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a","Type":"ContainerStarted","Data":"d577f6a44176c928a457869f4084228e47948fd10e226cd565ba592cb9372109"}
Nov 02 11:55:49.510967 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:49.510925    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt" event={"ID":"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a","Type":"ContainerDied","Data":"093f77f5f86f98f54a23fc84e74d6708a935539196949ccf3c4975bd721b24a6"}
Nov 02 11:55:49.510967 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:49.510959    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt" event={"ID":"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a","Type":"ContainerStarted","Data":"4caf7994efaa02cbee2988052486334c8b3962b94e01726ca8da0a64227081e9"}
Nov 02 11:55:50.515132 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:50.515099    2577 generic.go:356] "Generic (PLEG): container finished" podID="918aeea0-5975-4be5-a2f4-530e6c38f1ad" containerID="a9084391eb0aee5cb0b032051a2248af776405b178abeec5802ca8fdb027853c" exitCode=0
Nov 02 11:55:50.515566 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:50.515150    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gjgd" event={"ID":"918aeea0-5975-4be5-a2f4-530e6c38f1ad","Type":"ContainerDied","Data":"a9084391eb0aee5cb0b032051a2248af776405b178abeec5802ca8fdb027853c"}
Nov 02 11:55:51.352965 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:51.352923    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:51.352965 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:51.352956    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:51.353233 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:51.353042    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqwff" podUID="cbfec7e2-d13a-483b-8583-5f19481690c1"
Nov 02 11:55:51.353233 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:51.353109    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:51.353233 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:51.353226    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw29m" podUID="8161b19f-1a15-4f7d-9232-a9248995abf6"
Nov 02 11:55:51.353371 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:51.353312    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4hmk" podUID="2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab"
Nov 02 11:55:51.520282 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:51.520259    2577 generic.go:356] "Generic (PLEG): container finished" podID="918aeea0-5975-4be5-a2f4-530e6c38f1ad" containerID="5d8a7bbc5498a6f35a7cdd7eb09cd920aabf97700d2799fd9dbc283ea8c602a7" exitCode=0
Nov 02 11:55:51.520652 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:51.520507    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gjgd" event={"ID":"918aeea0-5975-4be5-a2f4-530e6c38f1ad","Type":"ContainerDied","Data":"5d8a7bbc5498a6f35a7cdd7eb09cd920aabf97700d2799fd9dbc283ea8c602a7"}
Nov 02 11:55:51.524662 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:51.524643    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 11:55:51.525029 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:51.525010    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt" event={"ID":"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a","Type":"ContainerStarted","Data":"1c892f11cc7a9d09743e0778d9dc947631c387dfffb599593cabe15d61ffbdbc"}
Nov 02 11:55:52.533046 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:52.530619    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gjgd" event={"ID":"918aeea0-5975-4be5-a2f4-530e6c38f1ad","Type":"ContainerStarted","Data":"a33d35c2bbacc6384b7d6f11255d4147711279e03dd96239f9137b6055b3e1d3"}
Nov 02 11:55:52.705046 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:52.705003    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4gjgd" podStartSLOduration=8.545807526 podStartE2EDuration="20.704990762s" totalImagesPullingTime="12.159183236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:52.704849049 +0000 UTC m=+20.925053807" watchObservedRunningTime="2026-11-02 11:55:52.704990762 +0000 UTC m=+20.925195511"
Nov 02 11:55:53.352794 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:53.352759    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:53.352794 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:53.352793    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:53.352794 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:53.352794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:53.353042 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:53.352866    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4hmk" podUID="2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab"
Nov 02 11:55:53.353042 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:53.352925    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw29m" podUID="8161b19f-1a15-4f7d-9232-a9248995abf6"
Nov 02 11:55:53.353042 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:53.353021    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqwff" podUID="cbfec7e2-d13a-483b-8583-5f19481690c1"
Nov 02 11:55:53.398523 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:53.398496    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k6kl8"
Nov 02 11:55:53.398660 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:53.398617    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 11:55:53.399072 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:53.399057    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k6kl8"
Nov 02 11:55:54.538273 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:54.538114    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 11:55:54.538824 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:54.538596    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt" event={"ID":"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a","Type":"ContainerStarted","Data":"943971759b8a76ee1e415f4ade2a13d405afbb6bb2e2aeb74920f23e5a61a2e0"}
Nov 02 11:55:54.538901 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:54.538851    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:54.538953 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:54.538918    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:54.539056 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:54.539039    2577 scope.go:122] "RemoveContainer" containerID="093f77f5f86f98f54a23fc84e74d6708a935539196949ccf3c4975bd721b24a6"
Nov 02 11:55:54.554589 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:54.554564    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:54.554823 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:54.554809    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:55:55.353319 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:55.353288    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:55.353319 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:55.353307    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:55.353559 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:55.353294    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:55.353559 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:55.353380    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqwff" podUID="cbfec7e2-d13a-483b-8583-5f19481690c1"
Nov 02 11:55:55.353559 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:55.353451    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw29m" podUID="8161b19f-1a15-4f7d-9232-a9248995abf6"
Nov 02 11:55:55.353559 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:55.353502    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4hmk" podUID="2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab"
Nov 02 11:55:55.543834 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:55.543808    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 11:55:55.544214 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:55.544170    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt" event={"ID":"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a","Type":"ContainerStarted","Data":"4c43545313a910782822ab884a3588a8f7a8bb3f37ed212dc425c6c1c24a0fc5"}
Nov 02 11:55:55.544275 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:55.544262    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 11:55:55.678697 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:55.678610    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt" podStartSLOduration=9.292807709 podStartE2EDuration="23.678597703s" totalImagesPullingTime="14.385789994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:55.678263965 +0000 UTC m=+23.898468905" watchObservedRunningTime="2026-11-02 11:55:55.678597703 +0000 UTC m=+23.898802460"
Nov 02 11:55:56.232186 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:56.232157    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sw29m"]
Nov 02 11:55:56.232395 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:56.232254    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:56.232395 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:56.232333    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw29m" podUID="8161b19f-1a15-4f7d-9232-a9248995abf6"
Nov 02 11:55:56.235263 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:56.235231    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bqwff"]
Nov 02 11:55:56.235400 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:56.235346    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:56.235516 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:56.235463    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqwff" podUID="cbfec7e2-d13a-483b-8583-5f19481690c1"
Nov 02 11:55:56.235815 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:56.235794    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x4hmk"]
Nov 02 11:55:56.235881 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:56.235867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:56.235990 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:56.235953    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4hmk" podUID="2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab"
Nov 02 11:55:56.546010 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:56.545924    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 11:55:58.352712 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:58.352682    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:55:58.353079 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:58.352682    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:55:58.353079 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:58.352794    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4hmk" podUID="2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab"
Nov 02 11:55:58.353079 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:55:58.352685    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:55:58.353079 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:58.352877    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqwff" podUID="cbfec7e2-d13a-483b-8583-5f19481690c1"
Nov 02 11:55:58.353079 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:55:58.352950    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw29m" podUID="8161b19f-1a15-4f7d-9232-a9248995abf6"
Nov 02 11:56:00.353045 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:00.353007    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:56:00.353045 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:00.353029    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:56:00.353469 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:00.353021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:56:00.353469 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:00.353126    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sw29m" podUID="8161b19f-1a15-4f7d-9232-a9248995abf6"
Nov 02 11:56:00.353469 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:00.353176    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x4hmk" podUID="2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab"
Nov 02 11:56:00.353469 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:00.353259    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bqwff" podUID="cbfec7e2-d13a-483b-8583-5f19481690c1"
Nov 02 11:56:01.733756 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:01.733584    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:01.734208 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:01.733957    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 11:56:01.744583 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:01.744559    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:02.010668 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.010635    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeReady"
Nov 02 11:56:02.010813 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.010778    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 11:56:02.107220 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.107191    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dc2vq"]
Nov 02 11:56:02.145876 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.145852    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-znfl5"]
Nov 02 11:56:02.146024 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.145935    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dc2vq"
Nov 02 11:56:02.181514 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.181483    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:02.188489 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.188468    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb-cert\") pod \"ingress-canary-dc2vq\" (UID: \"037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb\") " pod="openshift-ingress-canary/ingress-canary-dc2vq"
Nov 02 11:56:02.188566 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.188497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlr4l\" (UniqueName: \"kubernetes.io/projected/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb-kube-api-access-qlr4l\") pod \"ingress-canary-dc2vq\" (UID: \"037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb\") " pod="openshift-ingress-canary/ingress-canary-dc2vq"
Nov 02 11:56:02.191598 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.191574    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dc2vq"]
Nov 02 11:56:02.197260 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.197239    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-znfl5"]
Nov 02 11:56:02.211264 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.211246    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 11:56:02.211360 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.211278    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 11:56:02.235022 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.235008    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 11:56:02.260360 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.260340    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 11:56:02.260494 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.260480    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qcv25\""
Nov 02 11:56:02.260556 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.260492    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hqqr9\""
Nov 02 11:56:02.260556 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.260536    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 11:56:02.288856 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.288809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb-cert\") pod \"ingress-canary-dc2vq\" (UID: \"037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb\") " pod="openshift-ingress-canary/ingress-canary-dc2vq"
Nov 02 11:56:02.288856 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.288835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlr4l\" (UniqueName: \"kubernetes.io/projected/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb-kube-api-access-qlr4l\") pod \"ingress-canary-dc2vq\" (UID: \"037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb\") " pod="openshift-ingress-canary/ingress-canary-dc2vq"
Nov 02 11:56:02.288856 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.288855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-tmp-dir\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:02.289086 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.288901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-config-volume\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:02.289086 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.288918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:02.289086 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.288934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85sft\" (UniqueName: \"kubernetes.io/projected/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-kube-api-access-85sft\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:02.292454 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.292433    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb-cert\") pod \"ingress-canary-dc2vq\" (UID: \"037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb\") " pod="openshift-ingress-canary/ingress-canary-dc2vq"
Nov 02 11:56:02.350257 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.350239    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlr4l\" (UniqueName: \"kubernetes.io/projected/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb-kube-api-access-qlr4l\") pod \"ingress-canary-dc2vq\" (UID: \"037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb\") " pod="openshift-ingress-canary/ingress-canary-dc2vq"
Nov 02 11:56:02.353163 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.353145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:56:02.353351 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.353171    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:56:02.353462 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.353218    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:56:02.389410 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.389388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-config-volume\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:02.389496 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.389414    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:02.389496 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.389432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85sft\" (UniqueName: \"kubernetes.io/projected/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-kube-api-access-85sft\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:02.389496 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.389467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-tmp-dir\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:02.389600 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:02.389541    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 11:56:02.389646 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:02.389601    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls podName:19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:02.889583821 +0000 UTC m=+31.109788569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls") pod "dns-default-znfl5" (UID: "19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958") : secret "dns-default-metrics-tls" not found
Nov 02 11:56:02.389789 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.389772    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-tmp-dir\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:02.389918 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.389901    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-config-volume\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:02.407251 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.407233    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 11:56:02.407318 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.407233    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 11:56:02.407464 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.407449    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 11:56:02.407542 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.407461    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gfx2g\""
Nov 02 11:56:02.407542 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.407473    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-lk8z8\""
Nov 02 11:56:02.407542 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.407508    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 11:56:02.407542 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.407529    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hc67r\""
Nov 02 11:56:02.453871 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.453845    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85sft\" (UniqueName: \"kubernetes.io/projected/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-kube-api-access-85sft\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:02.491936 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.491915    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dc2vq"
Nov 02 11:56:02.892426 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.892358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:02.892751 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:02.892490    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 11:56:02.892751 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:02.892557    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls podName:19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:03.892542466 +0000 UTC m=+32.112747200 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls") pod "dns-default-znfl5" (UID: "19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958") : secret "dns-default-metrics-tls" not found
Nov 02 11:56:02.908369 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:02.908343    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dc2vq"]
Nov 02 11:56:02.924625 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:56:02.924602    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037bfa58_bb7c_4ec4_8cc3_a1c7e72afbfb.slice/crio-6914df4348e21abd2a1b1044019d3b4067e9dc99299f3d472969ae07914e4bda WatchSource:0}: Error finding container 6914df4348e21abd2a1b1044019d3b4067e9dc99299f3d472969ae07914e4bda: Status 404 returned error can't find the container with id 6914df4348e21abd2a1b1044019d3b4067e9dc99299f3d472969ae07914e4bda
Nov 02 11:56:03.561734 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:03.561700    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dc2vq" event={"ID":"037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb","Type":"ContainerStarted","Data":"6914df4348e21abd2a1b1044019d3b4067e9dc99299f3d472969ae07914e4bda"}
Nov 02 11:56:03.899756 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:03.899681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:03.900132 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:03.899794    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 11:56:03.900132 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:03.899848    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls podName:19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:05.899833331 +0000 UTC m=+34.120038066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls") pod "dns-default-znfl5" (UID: "19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958") : secret "dns-default-metrics-tls" not found
Nov 02 11:56:05.209738 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:05.209707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:56:05.210213 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:05.209764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:56:05.210213 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:05.209873    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 11:56:05.210213 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:05.209939    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs podName:cbfec7e2-d13a-483b-8583-5f19481690c1 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:37.209923234 +0000 UTC m=+65.430127968 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs") pod "network-metrics-daemon-bqwff" (UID: "cbfec7e2-d13a-483b-8583-5f19481690c1") : secret "metrics-daemon-secret" not found
Nov 02 11:56:05.212308 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:05.212276    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:56:05.310746 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:05.310704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srjx5\" (UniqueName: \"kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5\") pod \"network-check-target-x4hmk\" (UID: \"2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab\") " pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:56:05.313465 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:05.313438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srjx5\" (UniqueName: \"kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5\") pod \"network-check-target-x4hmk\" (UID: \"2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab\") " pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:56:05.371451 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:05.371422    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:56:05.379697 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:05.379670    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:56:05.769609 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:05.769525    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sw29m"]
Nov 02 11:56:05.780426 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:56:05.780389    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8161b19f_1a15_4f7d_9232_a9248995abf6.slice/crio-2737c087af23a46d5a6dfffc08ee4de24fea74dc47c8082a88b18de3c7369361 WatchSource:0}: Error finding container 2737c087af23a46d5a6dfffc08ee4de24fea74dc47c8082a88b18de3c7369361: Status 404 returned error can't find the container with id 2737c087af23a46d5a6dfffc08ee4de24fea74dc47c8082a88b18de3c7369361
Nov 02 11:56:05.820354 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:05.820331    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x4hmk"]
Nov 02 11:56:05.822362 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:56:05.822339    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d717e6f_b6a7_48e7_a04a_c90a78b0e5ab.slice/crio-298769ad3a548dd07b533dc0753bd4b41979cbf2a6a330916079a884fe2197d4 WatchSource:0}: Error finding container 298769ad3a548dd07b533dc0753bd4b41979cbf2a6a330916079a884fe2197d4: Status 404 returned error can't find the container with id 298769ad3a548dd07b533dc0753bd4b41979cbf2a6a330916079a884fe2197d4
Nov 02 11:56:05.916475 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:05.916451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:05.916589 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:05.916577    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 11:56:05.916635 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:05.916625    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls podName:19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:09.916613162 +0000 UTC m=+38.136817895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls") pod "dns-default-znfl5" (UID: "19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958") : secret "dns-default-metrics-tls" not found
Nov 02 11:56:06.569701 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:06.569664    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dc2vq" event={"ID":"037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb","Type":"ContainerStarted","Data":"2f94e4b377cdae017c48aa9bb4f9b12df730f324359a1e33cd6dcbdb4601c6b0"}
Nov 02 11:56:06.570847 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:06.570823    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x4hmk" event={"ID":"2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab","Type":"ContainerStarted","Data":"298769ad3a548dd07b533dc0753bd4b41979cbf2a6a330916079a884fe2197d4"}
Nov 02 11:56:06.571866 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:06.571842    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sw29m" event={"ID":"8161b19f-1a15-4f7d-9232-a9248995abf6","Type":"ContainerStarted","Data":"2737c087af23a46d5a6dfffc08ee4de24fea74dc47c8082a88b18de3c7369361"}
Nov 02 11:56:06.687048 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:06.687003    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dc2vq" podStartSLOduration=1.557281682 podStartE2EDuration="4.686990848s" totalImagesPullingTime="3.129709166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:06.68662093 +0000 UTC m=+34.906825689" watchObservedRunningTime="2026-11-02 11:56:06.686990848 +0000 UTC m=+34.907195598"
Nov 02 11:56:09.947252 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:09.947216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:09.947676 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:09.947331    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 11:56:09.947676 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:09.947390    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls podName:19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:17.947376245 +0000 UTC m=+46.167580994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls") pod "dns-default-znfl5" (UID: "19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958") : secret "dns-default-metrics-tls" not found
Nov 02 11:56:10.177845 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.177774    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t692p"]
Nov 02 11:56:10.207013 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.206961    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:10.239940 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.239907    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t692p"]
Nov 02 11:56:10.290072 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.290035    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 11:56:10.290547 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.290522    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fgj5q"]
Nov 02 11:56:10.314259 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.314237    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 11:56:10.314549 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.314530    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 11:56:10.314549 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.314552    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 11:56:10.314691 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.314588    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4wp8v\""
Nov 02 11:56:10.316056 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.316030    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fgj5q"
Nov 02 11:56:10.331210 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.331185    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fgj5q"]
Nov 02 11:56:10.349390 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.349367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2762d18-8915-4499-b4e8-4927bde39164-data-volume\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:10.349493 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.349421    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p99w5\" (UniqueName: \"kubernetes.io/projected/f2762d18-8915-4499-b4e8-4927bde39164-kube-api-access-p99w5\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:10.349538 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.349499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2762d18-8915-4499-b4e8-4927bde39164-crio-socket\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:10.349538 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.349527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:10.349607 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.349564    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:10.390056 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.390032    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 11:56:10.390166 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.390103    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 11:56:10.390166 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.390136    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-n2k42\""
Nov 02 11:56:10.416243 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.416222    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-vfccn-qnzjq"]
Nov 02 11:56:10.438069 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.438008    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:10.449969 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.449950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p99w5\" (UniqueName: \"kubernetes.io/projected/f2762d18-8915-4499-b4e8-4927bde39164-kube-api-access-p99w5\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:10.450074 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.450010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2762d18-8915-4499-b4e8-4927bde39164-crio-socket\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:10.450074 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.450037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:10.450152 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.450076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:10.450152 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.450094    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lql6t\" (UniqueName: \"kubernetes.io/projected/7c0e97ec-40c1-4554-8966-f2267a4b287c-kube-api-access-lql6t\") pod \"migrator-66df5d496b-fgj5q\" (UID: \"7c0e97ec-40c1-4554-8966-f2267a4b287c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fgj5q"
Nov 02 11:56:10.450152 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.450122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2762d18-8915-4499-b4e8-4927bde39164-data-volume\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:10.450401 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.450380    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2762d18-8915-4499-b4e8-4927bde39164-data-volume\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:10.450461 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:10.450382    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 11:56:10.450499 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:10.450467    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm podName:f2762d18-8915-4499-b4e8-4927bde39164 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:10.950448692 +0000 UTC m=+39.170653428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t692p" (UID: "f2762d18-8915-4499-b4e8-4927bde39164") : configmap "kube-rbac-proxy" not found
Nov 02 11:56:10.450557 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:10.450534    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 11:56:10.450557 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.450539    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2762d18-8915-4499-b4e8-4927bde39164-crio-socket\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:10.450637 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:10.450590    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls podName:f2762d18-8915-4499-b4e8-4927bde39164 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:10.950567161 +0000 UTC m=+39.170771913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t692p" (UID: "f2762d18-8915-4499-b4e8-4927bde39164") : secret "insights-runtime-extractor-tls" not found
Nov 02 11:56:10.476346 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.476324    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vfccn-qnzjq"]
Nov 02 11:56:10.491757 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.491739    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vv2bx\""
Nov 02 11:56:10.495018 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.494999    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 11:56:10.546709 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.546681    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 11:56:10.549211 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.549186    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p99w5\" (UniqueName: \"kubernetes.io/projected/f2762d18-8915-4499-b4e8-4927bde39164-kube-api-access-p99w5\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:10.551160 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.551136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lql6t\" (UniqueName: \"kubernetes.io/projected/7c0e97ec-40c1-4554-8966-f2267a4b287c-kube-api-access-lql6t\") pod \"migrator-66df5d496b-fgj5q\" (UID: \"7c0e97ec-40c1-4554-8966-f2267a4b287c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fgj5q"
Nov 02 11:56:10.551240 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.551211    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/96b40a95-1bfc-4388-b366-f5464623ddb5-archives-path\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:10.551288 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.551253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-service-ca-bundle\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:10.551370 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.551353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjrdq\" (UniqueName: \"kubernetes.io/projected/96b40a95-1bfc-4388-b366-f5464623ddb5-kube-api-access-vjrdq\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:10.551461 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.551446    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-trusted-ca-bundle\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:10.584379 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.584348    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x4hmk" event={"ID":"2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab","Type":"ContainerStarted","Data":"ff84bf063716fedb982185ec3bf3b80f2417fe649d748210fda075be0c917752"}
Nov 02 11:56:10.584538 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.584496    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:56:10.616025 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.616000    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lql6t\" (UniqueName: \"kubernetes.io/projected/7c0e97ec-40c1-4554-8966-f2267a4b287c-kube-api-access-lql6t\") pod \"migrator-66df5d496b-fgj5q\" (UID: \"7c0e97ec-40c1-4554-8966-f2267a4b287c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fgj5q"
Nov 02 11:56:10.641046 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.641012    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fgj5q"
Nov 02 11:56:10.652384 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.652354    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjrdq\" (UniqueName: \"kubernetes.io/projected/96b40a95-1bfc-4388-b366-f5464623ddb5-kube-api-access-vjrdq\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:10.652526 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.652402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-trusted-ca-bundle\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:10.652526 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.652472    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/96b40a95-1bfc-4388-b366-f5464623ddb5-archives-path\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:10.652526 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.652500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-service-ca-bundle\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:10.653156 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.653109    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-service-ca-bundle\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:10.653361 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.653342    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/96b40a95-1bfc-4388-b366-f5464623ddb5-archives-path\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:10.653782 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.653755    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-trusted-ca-bundle\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:10.715714 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.715684    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjrdq\" (UniqueName: \"kubernetes.io/projected/96b40a95-1bfc-4388-b366-f5464623ddb5-kube-api-access-vjrdq\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:10.772440 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.772400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:10.955418 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.955345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:10.955779 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:10.955417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:10.955779 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:10.955467    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 11:56:10.955779 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:10.955517    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 11:56:10.955779 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:10.955533    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm podName:f2762d18-8915-4499-b4e8-4927bde39164 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:11.955518214 +0000 UTC m=+40.175722967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t692p" (UID: "f2762d18-8915-4499-b4e8-4927bde39164") : configmap "kube-rbac-proxy" not found
Nov 02 11:56:10.955779 ip-10-0-133-239 kubenswrapper[2577]: E1102 11:56:10.955558    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls podName:f2762d18-8915-4499-b4e8-4927bde39164 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:11.955548071 +0000 UTC m=+40.175752804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t692p" (UID: "f2762d18-8915-4499-b4e8-4927bde39164") : secret "insights-runtime-extractor-tls" not found
Nov 02 11:56:11.024371 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:11.024323    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x4hmk" podStartSLOduration=35.081949761 podStartE2EDuration="39.024310574s" totalImagesPullingTime="3.942360813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:10.724813815 +0000 UTC m=+38.945018571" watchObservedRunningTime="2026-11-02 11:56:11.024310574 +0000 UTC m=+39.244515348"
Nov 02 11:56:11.025108 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:11.025091    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fgj5q"]
Nov 02 11:56:11.046839 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:56:11.046811    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0e97ec_40c1_4554_8966_f2267a4b287c.slice/crio-f26d54e310aa146f91efc4d25dabd69ec761480fa2bf918e0fb5d5918252ae79 WatchSource:0}: Error finding container f26d54e310aa146f91efc4d25dabd69ec761480fa2bf918e0fb5d5918252ae79: Status 404 returned error can't find the container with id f26d54e310aa146f91efc4d25dabd69ec761480fa2bf918e0fb5d5918252ae79
Nov 02 11:56:11.139724 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:11.139670    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vfccn-qnzjq"]
Nov 02 11:56:11.145996 ip-10-0-133-239 kubenswrapper[2577]: W1102 11:56:11.145949    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b40a95_1bfc_4388_b366_f5464623ddb5.slice/crio-5fa794ba481e5c1ece7bb9d2926631d05c08d79d5d4f6f8dd44c9e2217ffa826 WatchSource:0}: Error finding container 5fa794ba481e5c1ece7bb9d2926631d05c08d79d5d4f6f8dd44c9e2217ffa826: Status 404 returned error can't find the container with id 5fa794ba481e5c1ece7bb9d2926631d05c08d79d5d4f6f8dd44c9e2217ffa826
Nov 02 11:56:11.368244 ip-10-0-133-239 kubenswrapper[2577]: I1102 11:56:11.368165    2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 11:56:11.368325 ip-10-0-133-239 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 11:56:11.388377 ip-10-0-133-239 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 11:56:11.388546 ip-10-0-133-239 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 11:56:11.388593 ip-10-0-133-239 systemd[1]: kubelet.service: Consumed 2.169s CPU time.
Nov 02 11:56:11.407436 ip-10-0-133-239 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 11:56:11.501211 ip-10-0-133-239 kubenswrapper[4464]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 11:56:11.501211 ip-10-0-133-239 kubenswrapper[4464]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 11:56:11.501211 ip-10-0-133-239 kubenswrapper[4464]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 11:56:11.501211 ip-10-0-133-239 kubenswrapper[4464]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 11:56:11.503255 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503181    4464 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 11:56:11.503255 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503254    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 11:56:11.503255 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503258    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503262    4464 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503264    4464 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503267    4464 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503270    4464 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503273    4464 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503276    4464 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503279    4464 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503281    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503284    4464 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503287    4464 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503290    4464 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503292    4464 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503295    4464 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503298    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503301    4464 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503304    4464 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503306    4464 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503309    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503312    4464 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 11:56:11.503355 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503315    4464 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503317    4464 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503320    4464 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503323    4464 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503325    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503336    4464 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503339    4464 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503342    4464 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503344    4464 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503347    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503349    4464 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503354    4464 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503360    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503363    4464 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503366    4464 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503369    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503371    4464 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503374    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503377    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 11:56:11.503867 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503379    4464 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503382    4464 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503385    4464 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503387    4464 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503390    4464 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503393    4464 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503395    4464 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503398    4464 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503401    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503403    4464 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503406    4464 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503408    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503411    4464 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503413    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503416    4464 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503418    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503423    4464 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503426    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503429    4464 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 11:56:11.504346 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503432    4464 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503435    4464 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503437    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503440    4464 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503443    4464 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503446    4464 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503449    4464 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503452    4464 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503454    4464 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503457    4464 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503459    4464 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503462    4464 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503465    4464 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503467    4464 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503470    4464 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503473    4464 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503475    4464 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503478    4464 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503480    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503483    4464 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 11:56:11.504805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503485    4464 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503488    4464 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503491    4464 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503493    4464 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503496    4464 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503499    4464 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503501    4464 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503504    4464 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503506    4464 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503509    4464 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503511    4464 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503514    4464 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503517    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503521    4464 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503524    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503527    4464 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503530    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503533    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503536    4464 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503538    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 11:56:11.505337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503541    4464 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503543    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503546    4464 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503548    4464 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503551    4464 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.503553    4464 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504282    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504289    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504292    4464 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504295    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504298    4464 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504300    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504303    4464 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504306    4464 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504308    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504311    4464 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504313    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504317    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504320    4464 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 11:56:11.505855 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504323    4464 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504326    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504329    4464 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504331    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504334    4464 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504337    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504340    4464 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504343    4464 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504345    4464 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504348    4464 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504351    4464 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504354    4464 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504357    4464 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504359    4464 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504362    4464 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504364    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504367    4464 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504370    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504372    4464 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 11:56:11.506332 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504375    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504377    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504380    4464 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504382    4464 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504385    4464 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504387    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504390    4464 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504393    4464 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504396    4464 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504398    4464 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504401    4464 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504404    4464 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504406    4464 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504409    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504412    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504414    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504417    4464 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504420    4464 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504422    4464 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504425    4464 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 11:56:11.506803 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504428    4464 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504431    4464 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504433    4464 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504436    4464 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504439    4464 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504441    4464 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504444    4464 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504446    4464 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504448    4464 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504451    4464 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504454    4464 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504458    4464 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504461    4464 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504464    4464 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504466    4464 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504469    4464 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504471    4464 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504474    4464 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504476    4464 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504479    4464 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 11:56:11.507305 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504481    4464 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504483    4464 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504486    4464 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504489    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504492    4464 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504495    4464 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504498    4464 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504501    4464 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504504    4464 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504506    4464 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504509    4464 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504512    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504515    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504517    4464 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504520    4464 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504523    4464 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504526    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504529    4464 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504533    4464 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 11:56:11.507845 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504537    4464 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504540    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504543    4464 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504546    4464 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504548    4464 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504551    4464 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504554    4464 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504556    4464 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504559    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504562    4464 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504565    4464 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504567    4464 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504570    4464 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504572    4464 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.504575    4464 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505044    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505050    4464 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505053    4464 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505056    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 11:56:11.508327 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505058    4464 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505064    4464 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505067    4464 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505069    4464 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505072    4464 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505075    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505077    4464 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505080    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505083    4464 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505086    4464 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505089    4464 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505091    4464 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505094    4464 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505097    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505101    4464 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505105    4464 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505108    4464 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505111    4464 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505113    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 11:56:11.508785 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505116    4464 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505119    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505121    4464 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505124    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505127    4464 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505130    4464 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505132    4464 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505135    4464 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505137    4464 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505140    4464 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505143    4464 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505146    4464 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505148    4464 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505151    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505158    4464 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505160    4464 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505163    4464 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505166    4464 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505168    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505171    4464 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 11:56:11.509297 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505173    4464 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505176    4464 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505179    4464 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505182    4464 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505185    4464 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505187    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505190    4464 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505192    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505195    4464 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505198    4464 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505200    4464 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505203    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505205    4464 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505208    4464 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505210    4464 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505213    4464 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505216    4464 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505218    4464 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505221    4464 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505223    4464 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 11:56:11.509949 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505226    4464 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505228    4464 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505232    4464 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505234    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505237    4464 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505239    4464 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505243    4464 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505246    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505249    4464 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505252    4464 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505254    4464 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505257    4464 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505259    4464 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505262    4464 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505264    4464 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505268    4464 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505271    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505273    4464 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505276    4464 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505278    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 11:56:11.510667 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505281    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505284    4464 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505287    4464 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505289    4464 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505292    4464 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505295    4464 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505297    4464 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505299    4464 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505302    4464 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505305    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505307    4464 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505310    4464 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505312    4464 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505315    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505318    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505320    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505323    4464 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505325    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505329    4464 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 11:56:11.511188 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505332    4464 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505335    4464 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505340    4464 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.505344    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505433    4464 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505440    4464 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505447    4464 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505452    4464 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505456    4464 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505464    4464 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505473    4464 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505477    4464 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505481    4464 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505484    4464 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505487    4464 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505491    4464 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505494    4464 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505497    4464 flags.go:64] FLAG: --cgroup-root=""
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505500    4464 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505503    4464 flags.go:64] FLAG: --client-ca-file=""
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505506    4464 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505509    4464 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505513    4464 flags.go:64] FLAG: --cluster-domain=""
Nov 02 11:56:11.511709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505516    4464 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505519    4464 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505522    4464 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505525    4464 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505529    4464 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505532    4464 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505535    4464 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505538    4464 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505542    4464 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505545    4464 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505548    4464 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505551    4464 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505554    4464 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505558    4464 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505561    4464 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505564    4464 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505567    4464 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505570    4464 flags.go:64] FLAG: --enable-server="true"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505573    4464 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505578    4464 flags.go:64] FLAG: --event-burst="100"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505581    4464 flags.go:64] FLAG: --event-qps="50"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505587    4464 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505590    4464 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505593    4464 flags.go:64] FLAG: --eviction-hard=""
Nov 02 11:56:11.512273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505597    4464 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505600    4464 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505603    4464 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505606    4464 flags.go:64] FLAG: --eviction-soft=""
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505609    4464 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505612    4464 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505615    4464 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505618    4464 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505621    4464 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505623    4464 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505626    4464 flags.go:64] FLAG: --feature-gates=""
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505630    4464 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505633    4464 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505637    4464 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505640    4464 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505643    4464 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505647    4464 flags.go:64] FLAG: --help="false"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505650    4464 flags.go:64] FLAG: --hostname-override="ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505653    4464 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505657    4464 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505659    4464 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505663    4464 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505667    4464 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 11:56:11.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505670    4464 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505673    4464 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505676    4464 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505679    4464 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505682    4464 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505686    4464 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505691    4464 flags.go:64] FLAG: --kube-reserved=""
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505694    4464 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505697    4464 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505700    4464 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505703    4464 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505706    4464 flags.go:64] FLAG: --lock-file=""
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505709    4464 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505712    4464 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505715    4464 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505720    4464 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505723    4464 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505727    4464 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505730    4464 flags.go:64] FLAG: --logging-format="text"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505733    4464 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505736    4464 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505739    4464 flags.go:64] FLAG: --manifest-url=""
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505742    4464 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505746    4464 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505749    4464 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 11:56:11.513550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505753    4464 flags.go:64] FLAG: --max-pods="110"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505757    4464 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505760    4464 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505763    4464 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505766    4464 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505769    4464 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505772    4464 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505775    4464 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505781    4464 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505784    4464 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505787    4464 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505790    4464 flags.go:64] FLAG: --pod-cidr=""
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505793    4464 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505797    4464 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505801    4464 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505804    4464 flags.go:64] FLAG: --port="10250"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505807    4464 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505810    4464 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-096a8732a36243750"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505813    4464 flags.go:64] FLAG: --qos-reserved=""
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505816    4464 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505819    4464 flags.go:64] FLAG: --register-node="true"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505822    4464 flags.go:64] FLAG: --register-with-taints=""
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505826    4464 flags.go:64] FLAG: --registry-burst="10"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505829    4464 flags.go:64] FLAG: --registry-qps="5"
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505832    4464 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505835    4464 flags.go:64] FLAG: --reserved-memory=""
Nov 02 11:56:11.514167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505841    4464 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505844    4464 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505847    4464 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505850    4464 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505853    4464 flags.go:64] FLAG: --runonce="false"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505856    4464 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505860    4464 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505863    4464 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505866    4464 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505869    4464 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505872    4464 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505875    4464 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505878    4464 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505881    4464 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505884    4464 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505887    4464 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505890    4464 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505893    4464 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505896    4464 flags.go:64] FLAG: --system-cgroups=""
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505903    4464 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505909    4464 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505912    4464 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505915    4464 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505919    4464 flags.go:64] FLAG: --tls-min-version=""
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505922    4464 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 11:56:11.514852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505924    4464 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505927    4464 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505932    4464 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505935    4464 flags.go:64] FLAG: --v="2"
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505940    4464 flags.go:64] FLAG: --version="false"
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505944    4464 flags.go:64] FLAG: --vmodule=""
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505948    4464 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.505952    4464 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506090    4464 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506094    4464 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506096    4464 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506099    4464 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506102    4464 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506105    4464 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506108    4464 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506111    4464 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506113    4464 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506116    4464 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506119    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506122    4464 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506124    4464 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506127    4464 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 11:56:11.515495 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506130    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506132    4464 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506135    4464 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506137    4464 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506142    4464 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506144    4464 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506147    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506150    4464 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506153    4464 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506156    4464 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506158    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506161    4464 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506165    4464 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506167    4464 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506170    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506173    4464 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506175    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506178    4464 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506182    4464 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 11:56:11.516086 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506186    4464 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506189    4464 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506192    4464 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506195    4464 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506197    4464 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506200    4464 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506203    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506205    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506208    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506211    4464 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506213    4464 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506216    4464 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506218    4464 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506221    4464 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506223    4464 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506226    4464 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506229    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506233    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506235    4464 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 11:56:11.516724 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506238    4464 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506241    4464 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506244    4464 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506247    4464 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506249    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506252    4464 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506256    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506259    4464 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506261    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506264    4464 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506267    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506270    4464 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506273    4464 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506275    4464 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506278    4464 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506281    4464 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506284    4464 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506286    4464 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506289    4464 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 11:56:11.517287 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506291    4464 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506294    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506297    4464 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506300    4464 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506302    4464 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506305    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506308    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506311    4464 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506313    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506316    4464 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506319    4464 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506323    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506326    4464 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506328    4464 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506331    4464 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506335    4464 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506338    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506341    4464 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506345    4464 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 11:56:11.518108 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506350    4464 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 11:56:11.518808 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506353    4464 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 11:56:11.518808 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506356    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 11:56:11.518808 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506359    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 11:56:11.518808 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506362    4464 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 11:56:11.518808 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506364    4464 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 11:56:11.518808 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506367    4464 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 11:56:11.518808 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506370    4464 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 11:56:11.518808 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506372    4464 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 11:56:11.518808 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506375    4464 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 11:56:11.518808 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506377    4464 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 11:56:11.518808 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506380    4464 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 11:56:11.518808 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506383    4464 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 11:56:11.518808 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506385    4464 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 11:56:11.518808 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506388    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 11:56:11.518808 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506391    4464 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 11:56:11.518808 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.506416    4464 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506450    4464 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506454    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506457    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506460    4464 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506463    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506465    4464 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506468    4464 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506472    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506475    4464 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506477    4464 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506480    4464 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506483    4464 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506486    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506489    4464 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506493    4464 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506497    4464 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506502    4464 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506506    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 11:56:11.519240 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506509    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506512    4464 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506515    4464 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506517    4464 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506520    4464 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506523    4464 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506525    4464 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506528    4464 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506530    4464 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506533    4464 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506536    4464 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506538    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506541    4464 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506544    4464 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506546    4464 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506549    4464 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506552    4464 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506554    4464 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506557    4464 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506560    4464 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 11:56:11.519925 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506562    4464 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506567    4464 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506569    4464 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506572    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506574    4464 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506584    4464 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506587    4464 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506591    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506594    4464 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506596    4464 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506600    4464 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506603    4464 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506606    4464 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506608    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506626    4464 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506630    4464 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506633    4464 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506636    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506639    4464 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506642    4464 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 11:56:11.520513 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506644    4464 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506647    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506650    4464 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506653    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506655    4464 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506659    4464 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506663    4464 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506666    4464 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506669    4464 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506672    4464 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506674    4464 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506677    4464 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506680    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506684    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506686    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506689    4464 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506692    4464 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506694    4464 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506697    4464 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 11:56:11.521019 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506699    4464 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506702    4464 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506705    4464 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506711    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506715    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506717    4464 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506720    4464 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506722    4464 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506725    4464 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506728    4464 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506731    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506733    4464 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506736    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506739    4464 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506741    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506744    4464 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506747    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506749    4464 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506752    4464 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 11:56:11.521499 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506754    4464 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506757    4464 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506759    4464 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506763    4464 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506765    4464 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506768    4464 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506770    4464 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506774    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506777    4464 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.506779    4464 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.511669    4464 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.511685    4464 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.511705    4464 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.511710    4464 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511767    4464 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511773    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511777    4464 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511780    4464 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511782    4464 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511785    4464 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 11:56:11.522055 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511788    4464 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511791    4464 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511793    4464 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511797    4464 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511799    4464 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511802    4464 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511805    4464 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511808    4464 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511811    4464 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511814    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511816    4464 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511819    4464 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511822    4464 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511824    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511827    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511829    4464 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511832    4464 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511835    4464 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511838    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511841    4464 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 11:56:11.522839 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511844    4464 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511847    4464 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511851    4464 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511855    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511858    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511861    4464 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511863    4464 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511866    4464 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511870    4464 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511874    4464 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511878    4464 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511880    4464 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511883    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511886    4464 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511889    4464 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511892    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511895    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511898    4464 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 11:56:11.523388 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511901    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511903    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511906    4464 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511909    4464 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511912    4464 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511915    4464 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511917    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511920    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511922    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511925    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511928    4464 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511931    4464 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511933    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511936    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511939    4464 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511941    4464 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511944    4464 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511946    4464 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511949    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511952    4464 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 11:56:11.523843 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511954    4464 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511957    4464 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511960    4464 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511962    4464 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511965    4464 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511967    4464 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511970    4464 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511985    4464 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511988    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511991    4464 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511993    4464 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511996    4464 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.511998    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512001    4464 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512004    4464 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512007    4464 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512009    4464 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512012    4464 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512015    4464 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512017    4464 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 11:56:11.524337 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512020    4464 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512024    4464 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512027    4464 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512029    4464 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512032    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512035    4464 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512038    4464 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512041    4464 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512043    4464 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512046    4464 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512049    4464 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512051    4464 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512054    4464 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512057    4464 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512059    4464 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512062    4464 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512064    4464 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512067    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512070    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 11:56:11.524805 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512072    4464 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 11:56:11.525376 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512075    4464 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 11:56:11.525376 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512078    4464 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 11:56:11.525376 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.512104    4464 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 11:56:11.525376 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512247    4464 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 11:56:11.525376 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512254    4464 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 11:56:11.525376 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512259    4464 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 11:56:11.525376 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512262    4464 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 11:56:11.525376 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512265    4464 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 11:56:11.525376 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512268    4464 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 11:56:11.525376 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512271    4464 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 11:56:11.525376 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512274    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 11:56:11.525376 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512277    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 11:56:11.525376 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512280    4464 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 11:56:11.525376 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512283    4464 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 11:56:11.525376 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512286    4464 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 11:56:11.525376 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512288    4464 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512291    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512294    4464 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512297    4464 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512300    4464 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512302    4464 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512305    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512308    4464 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512310    4464 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512313    4464 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512316    4464 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512318    4464 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512321    4464 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512324    4464 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512326    4464 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512329    4464 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512332    4464 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512334    4464 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512337    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512340    4464 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 11:56:11.525789 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512342    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512345    4464 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512347    4464 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512351    4464 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512354    4464 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512356    4464 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512360    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512363    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512366    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512368    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512371    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512374    4464 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512376    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512379    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512382    4464 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512385    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512388    4464 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512390    4464 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512393    4464 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512396    4464 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 11:56:11.526284 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512399    4464 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512401    4464 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512404    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512406    4464 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512409    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512412    4464 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512415    4464 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512417    4464 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512420    4464 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512422    4464 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512425    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512427    4464 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512430    4464 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512432    4464 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512435    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512438    4464 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512441    4464 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512443    4464 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512446    4464 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512450    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 11:56:11.527205 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512452    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512455    4464 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512457    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512460    4464 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512463    4464 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512465    4464 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512468    4464 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512471    4464 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512473    4464 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512476    4464 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512479    4464 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512481    4464 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512484    4464 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512488    4464 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512491    4464 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512495    4464 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512498    4464 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512501    4464 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512503    4464 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 11:56:11.527888 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512506    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 11:56:11.528385 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512508    4464 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 11:56:11.528385 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512511    4464 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 11:56:11.528385 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512513    4464 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 11:56:11.528385 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512516    4464 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 11:56:11.528385 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512519    4464 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 11:56:11.528385 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512521    4464 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 11:56:11.528385 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512524    4464 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 11:56:11.528385 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512527    4464 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 11:56:11.528385 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512529    4464 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 11:56:11.528385 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512532    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 11:56:11.528385 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512534    4464 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 11:56:11.528385 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512537    4464 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 11:56:11.528385 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512540    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 11:56:11.528385 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512542    4464 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 11:56:11.528385 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.512565    4464 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 11:56:11.528385 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512594    4464 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512598    4464 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512602    4464 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512605    4464 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512608    4464 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512611    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512613    4464 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512616    4464 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512619    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512621    4464 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512624    4464 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512628    4464 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512631    4464 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512634    4464 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512637    4464 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512640    4464 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512643    4464 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512646    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512648    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 11:56:11.528834 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512651    4464 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512654    4464 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512657    4464 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512659    4464 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512662    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512667    4464 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512671    4464 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512674    4464 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512677    4464 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512679    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512682    4464 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512685    4464 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512688    4464 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512690    4464 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512693    4464 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512696    4464 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512698    4464 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512701    4464 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512704    4464 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512706    4464 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 11:56:11.529341 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512709    4464 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512712    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512714    4464 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512717    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512720    4464 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512722    4464 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512725    4464 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512727    4464 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512730    4464 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512733    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512735    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512738    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512740    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512744    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512746    4464 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512749    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512751    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512755    4464 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512760    4464 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 11:56:11.529862 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512763    4464 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512766    4464 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512769    4464 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512771    4464 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512774    4464 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512777    4464 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512780    4464 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512782    4464 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512785    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512787    4464 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512790    4464 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512793    4464 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512795    4464 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512798    4464 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512800    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512803    4464 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512806    4464 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512808    4464 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512811    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512814    4464 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 11:56:11.530422 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512817    4464 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512819    4464 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512822    4464 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512824    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512827    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512830    4464 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512832    4464 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512835    4464 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512838    4464 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512840    4464 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512844    4464 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512849    4464 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512852    4464 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512854    4464 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512857    4464 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512860    4464 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512863    4464 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512867    4464 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512871    4464 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 11:56:11.531007 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512874    4464 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 11:56:11.531526 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512877    4464 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 11:56:11.531526 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512880    4464 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 11:56:11.531526 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512882    4464 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 11:56:11.531526 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512885    4464 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 11:56:11.531526 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512888    4464 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 11:56:11.531526 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512890    4464 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 11:56:11.531526 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512893    4464 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 11:56:11.531526 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:11.512896    4464 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 11:56:11.531526 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.512965    4464 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 11:56:11.531526 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.513875    4464 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 11:56:11.531526 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.513925    4464 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 11:56:11.531526 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.514114    4464 server.go:1014] "Starting client certificate rotation"
Nov 02 11:56:11.531526 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.514256    4464 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 11:56:11.531526 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.514306    4464 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 11:50:32 +0000 UTC" deadline="2028-08-14 22:40:47.257626498 +0000 UTC"
Nov 02 11:56:11.531526 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.514346    4464 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15634h44m35.743284038s"
Nov 02 11:56:11.531526 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.514531    4464 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 11:56:11.531985 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.515607    4464 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 11:56:11.531985 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.516970    4464 log.go:25] "Validated CRI v1 runtime API"
Nov 02 11:56:11.531985 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.518753    4464 log.go:25] "Validated CRI v1 image API"
Nov 02 11:56:11.531985 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.519519    4464 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 11:56:11.531985 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.521001    4464 fs.go:135] Filesystem UUIDs: map[1553d9df-aa57-4184-9f7d-46e8ac04b35b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c7db4a47-c6bf-4840-a0f6-49dbdb8520ae:/dev/nvme0n1p3]
Nov 02 11:56:11.532137 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.521021    4464 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1284073a42cc36538dedb298665dddbb04d7b232c53be6480a5799a353fa4a85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1284073a42cc36538dedb298665dddbb04d7b232c53be6480a5799a353fa4a85/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1710e2a33d16ed503ec68bc7bec0797fdabee0474f2ccf93501753b3af0ec00f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1710e2a33d16ed503ec68bc7bec0797fdabee0474f2ccf93501753b3af0ec00f/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2588e069f31d8fed2b5d530cf6a7de8b128785c7699da05a74e4d57daa6c624a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2588e069f31d8fed2b5d530cf6a7de8b128785c7699da05a74e4d57daa6c624a/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2737c087af23a46d5a6dfffc08ee4de24fea74dc47c8082a88b18de3c7369361/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2737c087af23a46d5a6dfffc08ee4de24fea74dc47c8082a88b18de3c7369361/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/298769ad3a548dd07b533dc0753bd4b41979cbf2a6a330916079a884fe2197d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/298769ad3a548dd07b533dc0753bd4b41979cbf2a6a330916079a884fe2197d4/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58d7f8633a39d4d6aeb31881cd3fa5d9fac6663d0f08770b25f58d5da4c7833d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58d7f8633a39d4d6aeb31881cd3fa5d9fac6663d0f08770b25f58d5da4c7833d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5fa794ba481e5c1ece7bb9d2926631d05c08d79d5d4f6f8dd44c9e2217ffa826/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5fa794ba481e5c1ece7bb9d2926631d05c08d79d5d4f6f8dd44c9e2217ffa826/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6914df4348e21abd2a1b1044019d3b4067e9dc99299f3d472969ae07914e4bda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6914df4348e21abd2a1b1044019d3b4067e9dc99299f3d472969ae07914e4bda/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93fc077760a4de14212716c28c0a334546ef6478f75595d7c732c792ff7945c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93fc077760a4de14212716c28c0a334546ef6478f75595d7c732c792ff7945c4/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a26c9da442602e856b17686f6f9cc1907cecace0a33e514730133db50daf891a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a26c9da442602e856b17686f6f9cc1907cecace0a33e514730133db50daf891a/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a403d5c8339a18277bf2a6300d7fcd648ac158dd99dda437c4f8d268ef1ccff0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a403d5c8339a18277bf2a6300d7fcd648ac158dd99dda437c4f8d268ef1ccff0/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1e788a4fcc68154bacbfccaeaa4b363bab86ec9e0c805d63d666ac642e03106/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1e788a4fcc68154bacbfccaeaa4b363bab86ec9e0c805d63d666ac642e03106/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5632eeb6f484ba08f8999719e59c240ba6312687fdf7faa7115e8852adec820/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5632eeb6f484ba08f8999719e59c240ba6312687fdf7faa7115e8852adec820/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb316f99149fcb3de0f5cc3836c5732d066048fce2ee449c85a18fd411d1b1a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb316f99149fcb3de0f5cc3836c5732d066048fce2ee449c85a18fd411d1b1a6/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f26d54e310aa146f91efc4d25dabd69ec761480fa2bf918e0fb5d5918252ae79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f26d54e310aa146f91efc4d25dabd69ec761480fa2bf918e0fb5d5918252ae79/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb/volumes/kubernetes.io~projected/kube-api-access-qlr4l:{mountpoint:/var/lib/kubelet/pods/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb/volumes/kubernetes.io~projected/kube-api-access-qlr4l major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f7641b1-828f-46e8-b276-293aa6ab5615/volumes/kubernetes.io~projected/kube-api-access-rjlf7:{mountpoint:/var/lib/kubelet/pods/0f7641b1-828f-46e8-b276-293aa6ab5615/volumes/kubernetes.io~projected/kube-api-access-rjlf7 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958/volumes/kubernetes.io~projected/kube-api-access-85sft:{mountpoint:/var/lib/kubelet/pods/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958/volumes/kubernetes.io~projected/kube-api-access-85sft major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab/volumes/kubernetes.io~projected/kube-api-access-srjx5:{mountpoint:/var/lib/kubelet/pods/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab/volumes/kubernetes.io~projected/kube-api-access-srjx5 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3199abe8-4925-4fba-a291-860f9b7394b9/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3199abe8-4925-4fba-a291-860f9b7394b9/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3199abe8-4925-4fba-a291-860f9b7394b9/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3199abe8-4925-4fba-a291-860f9b7394b9/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3199abe8-4925-4fba-a291-860f9b7394b9/volumes/kubernetes.io~projected/kube-api-access-pmw6l:{mountpoint:/var/lib/kubelet/pods/3199abe8-4925-4fba-a291-860f9b7394b9/volumes/kubernetes.io~projected/kube-api-access-pmw6l major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5592809c-08dd-46c1-be01-d1da5d6d54b7/volumes/kubernetes.io~projected/kube-api-access-hhsj8:{mountpoint:/var/lib/kubelet/pods/5592809c-08dd-46c1-be01-d1da5d6d54b7/volumes/kubernetes.io~projected/kube-api-access-hhsj8 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/569eaf0a-207c-4686-ab02-82e91b45d285/volumes/kubernetes.io~projected/kube-api-access-snxcb:{mountpoint:/var/lib/kubelet/pods/569eaf0a-207c-4686-ab02-82e91b45d285/volumes/kubernetes.io~projected/kube-api-access-snxcb major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/751f0932-c6b4-4ee6-a4bf-f096e7d27e62/volumes/kubernetes.io~projected/kube-api-access-xxhnp:{mountpoint:/var/lib/kubelet/pods/751f0932-c6b4-4ee6-a4bf-f096e7d27e62/volumes/kubernetes.io~projected/kube-api-access-xxhnp major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c0e97ec-40c1-4554-8966-f2267a4b287c/volumes/kubernetes.io~projected/kube-api-access-lql6t:{mountpoint:/var/lib/kubelet/pods/7c0e97ec-40c1-4554-8966-f2267a4b287c/volumes/kubernetes.io~projected/kube-api-access-lql6t major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8161b19f-1a15-4f7d-9232-a9248995abf6/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8161b19f-1a15-4f7d-9232-a9248995abf6/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/836a65d8-536a-4183-9f71-4761970d5d15/volumes/kubernetes.io~projected/kube-api-access-t47mb:{mountpoint:/var/lib/kubelet/pods/836a65d8-536a-4183-9f71-4761970d5d15/volumes/kubernetes.io~projected/kube-api-access-t47mb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/918aeea0-5975-4be5-a2f4-530e6c38f1ad/volumes/kubernetes.io~projected/kube-api-access-tbbm7:{mountpoint:/var/lib/kubelet/pods/918aeea0-5975-4be5-a2f4-530e6c38f1ad/volumes/kubernetes.io~projected/kube-api-access-tbbm7 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96b40a95-1bfc-4388-b366-f5464623ddb5/volumes/kubernetes.io~projected/kube-api-access-vjrdq:{mountpoint:/var/lib/kubelet/pods/96b40a95-1bfc-4388-b366-f5464623ddb5/volumes/kubernetes.io~projected/kube-api-access-vjrdq major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/volumes/kubernetes.io~projected/kube-api-access-xdv8n:{mountpoint:/var/lib/kubelet/pods/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/volumes/kubernetes.io~projected/kube-api-access-xdv8n major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7058574-e988-4cc5-8a0c-fa961b9080c1/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b7058574-e988-4cc5-8a0c-fa961b9080c1/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbfec7e2-d13a-483b-8583-5f19481690c1/volumes/kubernetes.io~projected/kube-api-access-5lftk:{mountpoint:/var/lib/kubelet/pods/cbfec7e2-d13a-483b-8583-5f19481690c1/volumes/kubernetes.io~projected/kube-api-access-5lftk major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2762d18-8915-4499-b4e8-4927bde39164/volumes/kubernetes.io~projected/kube-api-access-p99w5:{mountpoint:/var/lib/kubelet/pods/f2762d18-8915-4499-b4e8-4927bde39164/volumes/kubernetes.io~projected/kube-api-access-p99w5 major:0 minor:224 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/45744f4cea93f5c62d46e489068c6fc0b71cbb045989c9a315e20acba2c2408a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6cc0556d38946d15e1d1793bd95dbddcb453d5b5fa55ffd7e8e34c98a189a068/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3c3aed7d5e2239c762f79b6f47f45ab676b39e21d8d9d6d488343199844b9703/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/5afaf0cff6cdac65b01c97a78e63ac40518976e09780a8431f6dfa4ed2f0490a/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/42c3e793ffab6b7a66adac960fa540ed913a29b6404d105718689c509e53f11b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/bc2a5f36b5202e74e93bbccac4a7e5f4aaabba8d52d6dca01c5b66a4df46c6f3/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-123:{mountpoint:/var/lib/containers/storage/overlay/061e15d5d2b36f56e37a03475e11029fe9cf78355199a230c0b37bf308597fe7/merged major:0 minor:123 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/acbefc20730ed8070ab206e5b6be04a12b4e110e6932592581e957eb0ed2ed3f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/4072d037795581b5f5a9951c88c0c130e34a9fea9799e45281dd54a8697184d2/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/5364eb86020f3c593f517ee91ba5e469a602d5110021b73cc6b03b255d8da2e4/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/14aa98f93eee39224cf8e54eac4acc9b0eba6b30de5f3d23709b5c246131b4c1/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/cced37a957580df69e9cf3d5e8b131f64535e3304ea104dd68e20fe16a34530e/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/251ccab0a48e48565e1ffa94e060a9e130423c1a0309532edc5029c31149044a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1b3a5b36bc6a080905978dcdaa94bd91fa50761652297b1f076c7dced813c9ab/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/f2e78064c15088fc9011053672478838abf36eada2596c484a00ddb2898d8ae5/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1d1e80167537b99d474ab276b56c93a94199f238e765d1c4087e3113a42f4b13/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/aa134e701a9137c71eb4d58e83283dd298ca52a2351abf7ec638ac599f6a25ea/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a8ed686b3ffed8159a1bb5ad95fbacd49c9fb61964032d4e5580e451a6304fa5/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/87f7b7f1b520601b048b8ebb59a09cfa880bfea57800ec2fccf5a3016d47fb8b/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/35a5e6ba7a1ab8458e6c94b95e619ec2bdd751fb6285f6ab6174e727108d0f95/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b2c5303c49dcfeab43edaa4a2501984eace21abe62544bf8395434a639b71235/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/04a3d8d9dde97ded659a0ed8f10040749a5806ac3a059141c873344dde251eda/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/8c1f96544a3ee64ab62f0e2fd33a3243df061e269a001d24510c9f16371b3636/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/ebc5b38a180f8e4c99cdc6a99962f9dadc4048cf9e2cc7e701d7c49018eea38c/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/c5f220f938e1c3b7d8e5a565e1fb472e9c533e3fb1dfae54b5673cfd4cec221c/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/e8decf34c14cd86d9a95c6bdb013f7daf58359efddc166f7e5b5cc1f7993c1b5/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/159df61974d571556f07efc860cff31624acfae33f59b09975209c5da2fb0dd6/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/6c1ed8ab448c49d80a1e8820fc03a10c45a746b4fbc79bfb936974517c1cdb42/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/96abc49d1485e04693d2e6e8e8524eda2ef0cffb9a8e12128238984ab7e17db4/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/6719876fe728000b6b9c5a1d3a33d279b22f8f483215de10bdcc332afe325658/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/75e5a97382fd1cf5e579d42b9c2eec68c198d2eeef48a5432761a78441bc0bda/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/0398d93fe1848af9825071077570321e174c430566eddad692033b26c6371099/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/fef55861e1e24a0fda12136a8b8130ec91b3e1c2ad12744c0209d150661c4f44/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/7cff10388ac013a47d28e3984b5e255c31f1872daf7034508663b0f0231323df/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/573b5ee954121edd3f4301382581ffa052fdeb566bdf76fe0c84bd4d402604c9/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/a3e82894c8a0ed8e1ad19fbd9a0c48a88702955eea81101b75928593e5618306/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/1a330fd391e53b20113c7f1a238c17d1feb02412bfe309559a48accc9e7e6df1/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/68129129bf2ec832a8c5ca02c4f81765694d4c1977b86104d859b36f708f7081/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/55bde0380daa56fd4487f186254c37f9a1c0d7d6bd17db8bbff599e2aac42d63/merged major:0 minor:98 fsType:overlay blockSize:0}]
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.527824    4464 manager.go:217] Machine: {Timestamp:2026-11-02 11:56:11.52686108 +0000 UTC m=+0.090569482 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099421 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d58a7fbab2e509e516dd25245ed50 SystemUUID:ec2d58a7-fbab-2e50-9e51-6dd25245ed50 BootID:3d55430b-4265-4e61-9b34-31b1587b356a Filesystems:[{Device:/var/lib/kubelet/pods/b7058574-e988-4cc5-8a0c-fa961b9080c1/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f26d54e310aa146f91efc4d25dabd69ec761480fa2bf918e0fb5d5918252ae79/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2737c087af23a46d5a6dfffc08ee4de24fea74dc47c8082a88b18de3c7369361/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1e788a4fcc68154bacbfccaeaa4b363bab86ec9e0c805d63d666ac642e03106/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1710e2a33d16ed503ec68bc7bec0797fdabee0474f2ccf93501753b3af0ec00f/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/836a65d8-536a-4183-9f71-4761970d5d15/volumes/kubernetes.io~projected/kube-api-access-t47mb DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/volumes/kubernetes.io~projected/kube-api-access-xdv8n DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-123 DeviceMajor:0 DeviceMinor:123 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb/volumes/kubernetes.io~projected/kube-api-access-qlr4l DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab/volumes/kubernetes.io~projected/kube-api-access-srjx5 DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f2762d18-8915-4499-b4e8-4927bde39164/volumes/kubernetes.io~projected/kube-api-access-p99w5 DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c0e97ec-40c1-4554-8966-f2267a4b287c/volumes/kubernetes.io~projected/kube-api-access-lql6t DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3199abe8-4925-4fba-a291-860f9b7394b9/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3199abe8-4925-4fba-a291-860f9b7394b9/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3199abe8-4925-4fba-a291-860f9b7394b9/volumes/kubernetes.io~projected/kube-api-access-pmw6l DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f7641b1-828f-46e8-b276-293aa6ab5615/volumes/kubernetes.io~projected/kube-api-access-rjlf7 DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1284073a42cc36538dedb298665dddbb04d7b232c53be6480a5799a353fa4a85/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8161b19f-1a15-4f7d-9232-a9248995abf6/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5632eeb6f484ba08f8999719e59c240ba6312687fdf7faa7115e8852adec820/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58d7f8633a39d4d6aeb31881cd3fa5d9fac6663d0f08770b25f58d5da4c7833d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958/volumes/kubernetes.io~projected/kube-api-access-85sft DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5fa794ba481e5c1ece7bb9d2926631d05c08d79d5d4f6f8dd44c9e2217ffa826/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/5592809c-08dd-46c1-be01-d1da5d6d54b7/volumes/kubernetes.io~projected/kube-api-access-hhsj8 DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/918aeea0-5975-4be5-a2f4-530e6c38f1ad/volumes/kubernetes.io~projected/kube-api-access-tbbm7 DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/298769ad3a548dd07b533dc0753bd4b41979cbf2a6a330916079a884fe2197d4/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbfec7e2-d13a-483b-8583-5f19481690c1/volumes/kubernetes.io~projected/kube-api-access-5lftk DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a403d5c8339a18277bf2a6300d7fcd648ac158dd99dda437c4f8d268ef1ccff0/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/751f0932-c6b4-4ee6-a4bf-f096e7d27e62/volumes/kubernetes.io~projected/kube-api-access-xxhnp DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/569eaf0a-207c-4686-ab02-82e91b45d285/volumes/kubernetes.io~projected/kube-api-access-snxcb DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93fc077760a4de14212716c28c0a334546ef6478f75595d7c732c792ff7945c4/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb316f99149fcb3de0f5cc3836c5732d066048fce2ee449c85a18fd411d1b1a6/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2588e069f31d8fed2b5d530cf6a7de8b128785c7699da05a74e4d57daa6c624a/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a26c9da442602e856b17686f6f9cc1907cecace0a33e514730133db50daf891a/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6914df4348e21abd2a1b1044019d3b4067e9dc99299f3d472969ae07914e4bda/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/96b40a95-1bfc-4388-b366-f5464623ddb5/volumes/kubernetes.io~projected/kube-api-access-vjrdq DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2737c087af23a46 MacAddress:d2:11:38:1e:39:b1 Speed:10000 Mtu:8901} {Name:298769ad3a548dd MacAddress:b6:42:91:23:ba:c5 Speed:10000 Mtu:8901} {Name:5fa794ba481e5c1 MacAddress:5a:d5:ec:15:e3:ac Speed:10000 Mtu:8901} {Name:6914df4348e21ab MacAddress:ee:8e:28:51:cc:0c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:01:48:5d:6c:77 Speed:0 Mtu:9001} {Name:br-int MacAddress:32:b7:ed:b3:34:23 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:01:48:5d:6c:77 Speed:0 Mtu:9001} {Name:f26d54e310aa146 MacAddress:92:21:44:7c:6e:dd Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:1e:94:8f:2a:41:96 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ea:58:1a:e2:45:d7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.528105    4464 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.528153    4464 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.528381    4464 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.528401    4464 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-239.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.528513    4464 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.528522    4464 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.528531    4464 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.528550    4464 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.528632    4464 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.528765    4464 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.528867    4464 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.528900    4464 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.528908    4464 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.528917    4464 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.528926    4464 kubelet.go:405] "Adding apiserver pod source"
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.528934    4464 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.529611    4464 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.529623    4464 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 11:56:11.532278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.530055    4464 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 11:56:11.536300 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536285    4464 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 11:56:11.536300 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536300    4464 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 11:56:11.536416 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536391    4464 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 11:56:11.536580 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536569    4464 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 11:56:11.536610 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536598    4464 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 11:56:11.536685 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536677    4464 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 11:56:11.536719 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536690    4464 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 11:56:11.536719 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536696    4464 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 11:56:11.536719 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536702    4464 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 11:56:11.536719 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536707    4464 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 11:56:11.536719 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536712    4464 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 11:56:11.536719 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536718    4464 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 11:56:11.536874 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536724    4464 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 11:56:11.536874 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536731    4464 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 11:56:11.536874 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536737    4464 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 11:56:11.536874 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536745    4464 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 11:56:11.536874 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536753    4464 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 11:56:11.536874 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536774    4464 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 11:56:11.536874 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.536780    4464 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 11:56:11.539919 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.539905    4464 server.go:1263] "Started kubelet"
Nov 02 11:56:11.540061 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.540012    4464 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 11:56:11.540128 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.540016    4464 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 11:56:11.540128 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.540122    4464 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 11:56:11.540426 ip-10-0-133-239 systemd[1]: Started Kubernetes Kubelet.
Nov 02 11:56:11.540591 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.540568    4464 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 11:56:11.541295 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.541280    4464 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 11:56:11.542620 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.542601    4464 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 11:56:11.542714 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.542687    4464 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 11:56:11.542714 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.542709    4464 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 11:56:11.542843 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.542721    4464 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 11:56:11.543117 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.543099    4464 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 11:56:11.543117 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:11.542706    4464 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-239.us-west-1.compute.internal\" not found"
Nov 02 11:56:11.543970 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.543929    4464 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 11:50:33 +0000 UTC" deadline="2028-07-27 08:48:52.06368778 +0000 UTC"
Nov 02 11:56:11.544079 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.543991    4464 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15188h52m40.519720381s"
Nov 02 11:56:11.545094 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545042    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f7641b1-828f-46e8-b276-293aa6ab5615" volumeName="kubernetes.io/empty-dir/0f7641b1-828f-46e8-b276-293aa6ab5615-tmp-dir" seLinuxMountContext=""
Nov 02 11:56:11.545195 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545100    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96b40a95-1bfc-4388-b366-f5464623ddb5" volumeName="kubernetes.io/projected/96b40a95-1bfc-4388-b366-f5464623ddb5-kube-api-access-vjrdq" seLinuxMountContext=""
Nov 02 11:56:11.545195 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545124    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958" volumeName="kubernetes.io/empty-dir/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-tmp-dir" seLinuxMountContext=""
Nov 02 11:56:11.545195 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545138    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3199abe8-4925-4fba-a291-860f9b7394b9" volumeName="kubernetes.io/projected/3199abe8-4925-4fba-a291-860f9b7394b9-kube-api-access-pmw6l" seLinuxMountContext=""
Nov 02 11:56:11.545195 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545152    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="569eaf0a-207c-4686-ab02-82e91b45d285" volumeName="kubernetes.io/projected/569eaf0a-207c-4686-ab02-82e91b45d285-kube-api-access-snxcb" seLinuxMountContext=""
Nov 02 11:56:11.545195 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545170    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d66d28c-8f94-4e4f-a07a-10b188c9bc6a" volumeName="kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 11:56:11.545195 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545183    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb" volumeName="kubernetes.io/projected/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb-kube-api-access-qlr4l" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545202    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3199abe8-4925-4fba-a291-860f9b7394b9" volumeName="kubernetes.io/empty-dir/3199abe8-4925-4fba-a291-860f9b7394b9-tmp" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545219    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="918aeea0-5975-4be5-a2f4-530e6c38f1ad" volumeName="kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545237    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d66d28c-8f94-4e4f-a07a-10b188c9bc6a" volumeName="kubernetes.io/secret/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545251    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7058574-e988-4cc5-8a0c-fa961b9080c1" volumeName="kubernetes.io/secret/b7058574-e988-4cc5-8a0c-fa961b9080c1-agent-certs" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545264    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f7641b1-828f-46e8-b276-293aa6ab5615" volumeName="kubernetes.io/projected/0f7641b1-828f-46e8-b276-293aa6ab5615-kube-api-access-rjlf7" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545282    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab" volumeName="kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545296    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="751f0932-c6b4-4ee6-a4bf-f096e7d27e62" volumeName="kubernetes.io/configmap/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-iptables-alerter-script" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545318    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c0e97ec-40c1-4554-8966-f2267a4b287c" volumeName="kubernetes.io/projected/7c0e97ec-40c1-4554-8966-f2267a4b287c-kube-api-access-lql6t" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545332    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="918aeea0-5975-4be5-a2f4-530e6c38f1ad" volumeName="kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cni-binary-copy" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545350    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96b40a95-1bfc-4388-b366-f5464623ddb5" volumeName="kubernetes.io/empty-dir/96b40a95-1bfc-4388-b366-f5464623ddb5-archives-path" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545362    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d66d28c-8f94-4e4f-a07a-10b188c9bc6a" volumeName="kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-env-overrides" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545374    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836a65d8-536a-4183-9f71-4761970d5d15" volumeName="kubernetes.io/configmap/836a65d8-536a-4183-9f71-4761970d5d15-multus-daemon-config" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545392    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="918aeea0-5975-4be5-a2f4-530e6c38f1ad" volumeName="kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545405    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96b40a95-1bfc-4388-b366-f5464623ddb5" volumeName="kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-service-ca-bundle" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545422    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96b40a95-1bfc-4388-b366-f5464623ddb5" volumeName="kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545438    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7058574-e988-4cc5-8a0c-fa961b9080c1" volumeName="kubernetes.io/configmap/b7058574-e988-4cc5-8a0c-fa961b9080c1-konnectivity-ca" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545451    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958" volumeName="kubernetes.io/configmap/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-config-volume" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545495    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="569eaf0a-207c-4686-ab02-82e91b45d285" volumeName="kubernetes.io/configmap/569eaf0a-207c-4686-ab02-82e91b45d285-serviceca" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545509    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836a65d8-536a-4183-9f71-4761970d5d15" volumeName="kubernetes.io/configmap/836a65d8-536a-4183-9f71-4761970d5d15-cni-binary-copy" seLinuxMountContext=""
Nov 02 11:56:11.545522 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545526    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836a65d8-536a-4183-9f71-4761970d5d15" volumeName="kubernetes.io/projected/836a65d8-536a-4183-9f71-4761970d5d15-kube-api-access-t47mb" seLinuxMountContext=""
Nov 02 11:56:11.546475 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545540    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d66d28c-8f94-4e4f-a07a-10b188c9bc6a" volumeName="kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovnkube-config" seLinuxMountContext=""
Nov 02 11:56:11.546475 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545566    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbfec7e2-d13a-483b-8583-5f19481690c1" volumeName="kubernetes.io/projected/cbfec7e2-d13a-483b-8583-5f19481690c1-kube-api-access-5lftk" seLinuxMountContext=""
Nov 02 11:56:11.546475 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545579    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2762d18-8915-4499-b4e8-4927bde39164" volumeName="kubernetes.io/empty-dir/f2762d18-8915-4499-b4e8-4927bde39164-data-volume" seLinuxMountContext=""
Nov 02 11:56:11.546475 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545596    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958" volumeName="kubernetes.io/projected/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-kube-api-access-85sft" seLinuxMountContext=""
Nov 02 11:56:11.546475 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.545611    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3199abe8-4925-4fba-a291-860f9b7394b9" volumeName="kubernetes.io/empty-dir/3199abe8-4925-4fba-a291-860f9b7394b9-etc-tuned" seLinuxMountContext=""
Nov 02 11:56:11.548043 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.548019    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="751f0932-c6b4-4ee6-a4bf-f096e7d27e62" volumeName="kubernetes.io/projected/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-kube-api-access-xxhnp" seLinuxMountContext=""
Nov 02 11:56:11.548143 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.548045    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8161b19f-1a15-4f7d-9232-a9248995abf6" volumeName="kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret" seLinuxMountContext=""
Nov 02 11:56:11.548143 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.548067    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d66d28c-8f94-4e4f-a07a-10b188c9bc6a" volumeName="kubernetes.io/projected/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-kube-api-access-xdv8n" seLinuxMountContext=""
Nov 02 11:56:11.548143 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.547829    4464 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 11:56:11.549009 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.548610    4464 factory.go:55] Registering systemd factory
Nov 02 11:56:11.549009 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.548632    4464 factory.go:223] Registration of the systemd container factory successfully
Nov 02 11:56:11.549009 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.548080    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2762d18-8915-4499-b4e8-4927bde39164" volumeName="kubernetes.io/projected/f2762d18-8915-4499-b4e8-4927bde39164-kube-api-access-p99w5" seLinuxMountContext=""
Nov 02 11:56:11.549009 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.548694    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5592809c-08dd-46c1-be01-d1da5d6d54b7" volumeName="kubernetes.io/projected/5592809c-08dd-46c1-be01-d1da5d6d54b7-kube-api-access-hhsj8" seLinuxMountContext=""
Nov 02 11:56:11.549009 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.548712    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="918aeea0-5975-4be5-a2f4-530e6c38f1ad" volumeName="kubernetes.io/projected/918aeea0-5975-4be5-a2f4-530e6c38f1ad-kube-api-access-tbbm7" seLinuxMountContext=""
Nov 02 11:56:11.549009 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.548725    4464 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb" volumeName="kubernetes.io/secret/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb-cert" seLinuxMountContext=""
Nov 02 11:56:11.549009 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.548736    4464 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 11:56:11.549009 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.548744    4464 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 11:56:11.549400 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.549339    4464 factory.go:153] Registering CRI-O factory
Nov 02 11:56:11.549400 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.549358    4464 factory.go:223] Registration of the crio container factory successfully
Nov 02 11:56:11.549400 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.549376    4464 factory.go:103] Registering Raw factory
Nov 02 11:56:11.549400 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.549390    4464 manager.go:1199] Started watching for new ooms in manager
Nov 02 11:56:11.549718 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.549703    4464 manager.go:319] Starting recovery of all containers
Nov 02 11:56:11.555107 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.555090    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 11:56:11.555594 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.555578    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 11:56:11.572971 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.572938    4464 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 11:56:11.574181 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.574161    4464 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 11:56:11.574264 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.574191    4464 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 11:56:11.574264 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.574212    4464 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 11:56:11.574264 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:11.574252    4464 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 11:56:11.588172 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.588151    4464 generic.go:356] "Generic (PLEG): container finished" podID="918aeea0-5975-4be5-a2f4-530e6c38f1ad" containerID="5d8a7bbc5498a6f35a7cdd7eb09cd920aabf97700d2799fd9dbc283ea8c602a7" exitCode=0
Nov 02 11:56:11.588172 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.588174    4464 generic.go:356] "Generic (PLEG): container finished" podID="918aeea0-5975-4be5-a2f4-530e6c38f1ad" containerID="a9084391eb0aee5cb0b032051a2248af776405b178abeec5802ca8fdb027853c" exitCode=0
Nov 02 11:56:11.588289 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.588181    4464 generic.go:356] "Generic (PLEG): container finished" podID="918aeea0-5975-4be5-a2f4-530e6c38f1ad" containerID="7494acb94f3bd30f567f9f8571991f2d72b4378813a09a542ebbf5de6d5b579a" exitCode=0
Nov 02 11:56:11.588289 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.588189    4464 generic.go:356] "Generic (PLEG): container finished" podID="918aeea0-5975-4be5-a2f4-530e6c38f1ad" containerID="869ad29dcf3e35a22b4ba4234d7f801e7bf374d48afaa20ccbf26a937f60e733" exitCode=0
Nov 02 11:56:11.588289 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.588198    4464 generic.go:356] "Generic (PLEG): container finished" podID="918aeea0-5975-4be5-a2f4-530e6c38f1ad" containerID="3aa81d288a6de43b89a77559018a68f13a49785353de96bb5d271c462e87ed23" exitCode=0
Nov 02 11:56:11.588289 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.588205    4464 generic.go:356] "Generic (PLEG): container finished" podID="918aeea0-5975-4be5-a2f4-530e6c38f1ad" containerID="9c0ae90acb31cee84c8d6a4fc25ccde6b48f747b2fd1115afa0aaecdec8dffdb" exitCode=0
Nov 02 11:56:11.599335 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.599315    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 11:56:11.599630 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.599604    4464 generic.go:356] "Generic (PLEG): container finished" podID="9d66d28c-8f94-4e4f-a07a-10b188c9bc6a" containerID="093f77f5f86f98f54a23fc84e74d6708a935539196949ccf3c4975bd721b24a6" exitCode=1
Nov 02 11:56:11.602857 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.602831    4464 generic.go:356] "Generic (PLEG): container finished" podID="fe93821f99a8c26257f227cd244c0afa" containerID="63fe7b8c30079dd31ab6b81e61b14ebd354990a43eac84e2b217687acbf12bf2" exitCode=0
Nov 02 11:56:11.612542 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.612514    4464 manager.go:324] Recovery completed
Nov 02 11:56:11.625071 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.625023    4464 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 11:56:11.626054 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.626038    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 11:56:11.626128 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.626071    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 11:56:11.626128 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.626086    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 11:56:11.627349 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.627329    4464 cpu_manager.go:226] "Starting" policy="none"
Nov 02 11:56:11.627349 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.627344    4464 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 11:56:11.627453 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.627345    4464 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 11:56:11.627453 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.627415    4464 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 11:56:11.627576 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.627561    4464 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 11:56:11.627619 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.627576    4464 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 11:56:11.627619 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.627592    4464 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 11:56:11.627619 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.627598    4464 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 11:56:11.627619 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.627603    4464 policy_none.go:50] "Start"
Nov 02 11:56:11.627619 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.627612    4464 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 11:56:11.627747 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.627624    4464 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 11:56:11.627799 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.627786    4464 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 11:56:11.627827 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.627805    4464 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 11:56:11.627827 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.627814    4464 policy_none.go:44] "Start"
Nov 02 11:56:11.628202 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.628187    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 11:56:11.628275 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.628213    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 11:56:11.628275 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.628223    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 11:56:11.631650 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.631629    4464 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 11:56:11.634609 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.634594    4464 manager.go:341] "Starting Device Plugin manager"
Nov 02 11:56:11.634688 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:11.634617    4464 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 11:56:11.634688 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.634625    4464 server.go:87] "Starting device plugin registration server"
Nov 02 11:56:11.634917 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.634903    4464 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 11:56:11.635006 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.634919    4464 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 11:56:11.635006 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.634994    4464 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 11:56:11.635117 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.635084    4464 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 11:56:11.635168 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.635145    4464 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 11:56:11.635168 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.635153    4464 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 11:56:11.635292 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.635217    4464 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T11:56:11.635116125Z","UUID":"f90ccde8-79dd-451c-aa29-7b5e6399a781","Handler":null,"Name":"","Endpoint":""}
Nov 02 11:56:11.636279 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:11.636251    4464 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 11:56:11.636363 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:11.636289    4464 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-239.us-west-1.compute.internal\" not found"
Nov 02 11:56:11.636490 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.636472    4464 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 11:56:11.636546 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.636518    4464 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 11:56:11.675379 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.675345    4464 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"]
Nov 02 11:56:11.675456 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.675446    4464 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 11:56:11.676290 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.676274    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 11:56:11.676375 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.676300    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 11:56:11.676375 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.676310    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 11:56:11.676519 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.676509    4464 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 11:56:11.676653 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.676639    4464 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 11:56:11.677228 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.677213    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 11:56:11.677303 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.677229    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 11:56:11.677303 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.677239    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 11:56:11.677303 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.677251    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 11:56:11.677303 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.677254    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 11:56:11.677303 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.677261    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 11:56:11.677452 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:11.677372    4464 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-239.us-west-1.compute.internal\" not found" node="ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:11.677545 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.677520    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal" event={"ID":"fe93821f99a8c26257f227cd244c0afa","Type":"ContainerStarted","Data":"3495aff3a966668a9ac0523e0320d66ec8c82831844e46c61deffe02f10d7387"}
Nov 02 11:56:11.677578 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.677555    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal" event={"ID":"fe93821f99a8c26257f227cd244c0afa","Type":"ContainerDied","Data":"63fe7b8c30079dd31ab6b81e61b14ebd354990a43eac84e2b217687acbf12bf2"}
Nov 02 11:56:11.677578 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.677573    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal" event={"ID":"fe93821f99a8c26257f227cd244c0afa","Type":"ContainerStarted","Data":"b5632eeb6f484ba08f8999719e59c240ba6312687fdf7faa7115e8852adec820"}
Nov 02 11:56:11.677633 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.677578    4464 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 11:56:11.677633 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.677588    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal" event={"ID":"1797323b44d1a789ea358554c2680a4d","Type":"ContainerStarted","Data":"73874b9877ed0d0c950db122b3aff86d43fe7578d9cfe2851b5e96d8d7f22d2a"}
Nov 02 11:56:11.677633 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.677599    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal" event={"ID":"1797323b44d1a789ea358554c2680a4d","Type":"ContainerStarted","Data":"1710e2a33d16ed503ec68bc7bec0797fdabee0474f2ccf93501753b3af0ec00f"}
Nov 02 11:56:11.678299 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.678285    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 11:56:11.678371 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.678313    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 11:56:11.678371 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.678328    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 11:56:11.678485 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:11.678471    4464 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-239.us-west-1.compute.internal\" not found" node="ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:11.735408 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.735388    4464 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 11:56:11.737671 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.737651    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 02 11:56:11.737754 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.737680    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 11:56:11.737754 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.737692    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 02 11:56:11.737754 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.737714    4464 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:11.746529 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.746508    4464 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 11:56:11.746605 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.746548    4464 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 11:56:11.746657 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.746640    4464 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 11:56:11.793339 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.793317    4464 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 11:56:11.797956 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.797941    4464 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 11:56:11.843743 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.843718    4464 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:11.847401 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.847386    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe93821f99a8c26257f227cd244c0afa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"fe93821f99a8c26257f227cd244c0afa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:11.847451 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.847410    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe93821f99a8c26257f227cd244c0afa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"fe93821f99a8c26257f227cd244c0afa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:11.847451 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.847427    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1797323b44d1a789ea358554c2680a4d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"1797323b44d1a789ea358554c2680a4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:11.854805 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.854788    4464 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:11.854862 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.854831    4464 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:11.855228 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.855210    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeNotReady"
Nov 02 11:56:11.855299 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.855234    4464 setters.go:546] "Node became not ready" node="ip-10-0-133-239.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T11:56:11Z","lastTransitionTime":"2026-11-02T11:56:11Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 02 11:56:11.896731 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.896686    4464 nodeinfomanager.go:471] existing CSINode "ip-10-0-133-239.us-west-1.compute.internal" is owned by different node (oldNodeID="1acd4bc0-9490-4c1e-972a-fbaa99521426", newNodeID=""), cleaning up...
Nov 02 11:56:11.905882 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.905869    4464 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 11:56:11.905932 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:11.905901    4464 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:11.905932 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.905912    4464 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:11.948030 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.948006    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1797323b44d1a789ea358554c2680a4d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"1797323b44d1a789ea358554c2680a4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:11.948118 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.947952    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1797323b44d1a789ea358554c2680a4d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"1797323b44d1a789ea358554c2680a4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:11.948118 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.948075    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe93821f99a8c26257f227cd244c0afa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"fe93821f99a8c26257f227cd244c0afa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:11.948118 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.948101    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe93821f99a8c26257f227cd244c0afa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"fe93821f99a8c26257f227cd244c0afa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:11.948246 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.948152    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe93821f99a8c26257f227cd244c0afa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"fe93821f99a8c26257f227cd244c0afa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:11.948246 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:11.948161    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe93821f99a8c26257f227cd244c0afa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal\" (UID: \"fe93821f99a8c26257f227cd244c0afa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:12.011325 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.011305    4464 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 11:56:12.011412 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:12.011340    4464 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:12.224614 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.224593    4464 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-239.us-west-1.compute.internal" event="NodeReady"
Nov 02 11:56:12.224751 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.224659    4464 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 11:56:12.529452 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.529383    4464 apiserver.go:52] "Watching apiserver"
Nov 02 11:56:12.609677 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.609657    4464 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:12.609811 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.609795    4464 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:12.686646 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.686619    4464 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 11:56:12.686646 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.686633    4464 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 11:56:12.686861 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.686663    4464 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 11:56:12.686861 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:12.686676    4464 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:12.686861 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:12.686675    4464 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal"
Nov 02 11:56:12.687167 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.687148    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-239.us-west-1.compute.internal","kube-system/global-pull-secret-syncer-sw29m","openshift-image-registry/node-ca-ddszj","openshift-insights/periodic-gathering-vfccn-qnzjq","kube-system/konnectivity-agent-k6kl8","openshift-cluster-node-tuning-operator/tuned-wdmr4","openshift-multus/network-metrics-daemon-bqwff","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7","openshift-dns/node-resolver-b4wmw","kube-system/kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal","openshift-service-ca/service-ca-5fffc97bdf-gffn9","openshift-insights/insights-runtime-extractor-t692p","openshift-multus/multus-7wp5n","openshift-ovn-kubernetes/ovnkube-node-w62gt","openshift-multus/multus-additional-cni-plugins-4gjgd","openshift-ingress-canary/ingress-canary-dc2vq","openshift-kube-storage-version-migrator/migrator-66df5d496b-fgj5q","openshift-network-operator/iptables-alerter-5zprr","openshift-network-diagnostics/network-check-target-x4hmk","openshift-dns/dns-default-znfl5"]
Nov 02 11:56:12.687567 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.687551    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:56:12.693857 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.693838    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:12.694239 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.694223    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:12.716617 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.716596    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-gffn9"
Nov 02 11:56:12.741612 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.741592    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 11:56:12.741612 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.741595    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gfx2g\""
Nov 02 11:56:12.741702 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.741634    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 11:56:12.744161 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.744141    4464 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 11:56:12.746948 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.746935    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 11:56:12.753093 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753079    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/96b40a95-1bfc-4388-b366-f5464623ddb5-archives-path\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:12.753152 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753101    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-host\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.753152 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753118    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t47mb\" (UniqueName: \"kubernetes.io/projected/836a65d8-536a-4183-9f71-4761970d5d15-kube-api-access-t47mb\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.753152 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753133    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-iptables-alerter-script\") pod \"iptables-alerter-5zprr\" (UID: \"751f0932-c6b4-4ee6-a4bf-f096e7d27e62\") " pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:56:12.753152 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753150    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-registration-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.753294 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753254    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/96b40a95-1bfc-4388-b366-f5464623ddb5-archives-path\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:12.753294 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753263    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.753379 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753360    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6e44ac17-b852-412a-bc80-5c3d3973d7ed-signing-key\") pod \"service-ca-5fffc97bdf-gffn9\" (UID: \"6e44ac17-b852-412a-bc80-5c3d3973d7ed\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gffn9"
Nov 02 11:56:12.753412 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753387    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-sys\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.753412 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753407    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b7058574-e988-4cc5-8a0c-fa961b9080c1-agent-certs\") pod \"konnectivity-agent-k6kl8\" (UID: \"b7058574-e988-4cc5-8a0c-fa961b9080c1\") " pod="kube-system/konnectivity-agent-k6kl8"
Nov 02 11:56:12.753464 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753422    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/569eaf0a-207c-4686-ab02-82e91b45d285-serviceca\") pod \"node-ca-ddszj\" (UID: \"569eaf0a-207c-4686-ab02-82e91b45d285\") " pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:56:12.753464 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753443    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-log-socket\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.753519 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753471    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-cni-netd\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.753547 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753532    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-systemd\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.753583 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753571    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f7641b1-828f-46e8-b276-293aa6ab5615-tmp-dir\") pod \"node-resolver-b4wmw\" (UID: \"0f7641b1-828f-46e8-b276-293aa6ab5615\") " pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:56:12.753612 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753594    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjlf7\" (UniqueName: \"kubernetes.io/projected/0f7641b1-828f-46e8-b276-293aa6ab5615-kube-api-access-rjlf7\") pod \"node-resolver-b4wmw\" (UID: \"0f7641b1-828f-46e8-b276-293aa6ab5615\") " pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:56:12.753645 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753613    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.753686 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753668    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f7641b1-828f-46e8-b276-293aa6ab5615-tmp-dir\") pod \"node-resolver-b4wmw\" (UID: \"0f7641b1-828f-46e8-b276-293aa6ab5615\") " pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:56:12.753722 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753684    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-systemd-units\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.753722 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753712    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-run-ovn-kubernetes\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.753808 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753761    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-cni-bin\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.753867 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753803    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lql6t\" (UniqueName: \"kubernetes.io/projected/7c0e97ec-40c1-4554-8966-f2267a4b287c-kube-api-access-lql6t\") pod \"migrator-66df5d496b-fgj5q\" (UID: \"7c0e97ec-40c1-4554-8966-f2267a4b287c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fgj5q"
Nov 02 11:56:12.753999 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.753961    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-multus-certs\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.754082 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754007    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2762d18-8915-4499-b4e8-4927bde39164-crio-socket\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:12.754082 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754033    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srjx5\" (UniqueName: \"kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5\") pod \"network-check-target-x4hmk\" (UID: \"2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab\") " pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:56:12.754082 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754071    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-etc-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.754191 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754093    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-trusted-ca-bundle\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:12.754191 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754156    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-var-lib-kubelet\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.754251 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754194    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-netns\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.754251 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754216    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovnkube-script-lib\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.754251 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754234    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysconfig\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.754342 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754267    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-tmp-dir\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:12.754342 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754334    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-socket-dir-parent\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.754512 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754368    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-tmp-dir\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:12.754512 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754422    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/836a65d8-536a-4183-9f71-4761970d5d15-multus-daemon-config\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.754512 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754449    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:56:12.754512 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754474    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbbm7\" (UniqueName: \"kubernetes.io/projected/918aeea0-5975-4be5-a2f4-530e6c38f1ad-kube-api-access-tbbm7\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:56:12.754665 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754519    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-device-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.754665 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754543    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lftk\" (UniqueName: \"kubernetes.io/projected/cbfec7e2-d13a-483b-8583-5f19481690c1-kube-api-access-5lftk\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:56:12.754665 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754567    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysctl-d\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.754665 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754610    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85sft\" (UniqueName: \"kubernetes.io/projected/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-kube-api-access-85sft\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:12.754841 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754663    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-conf-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.754841 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754706    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/836a65d8-536a-4183-9f71-4761970d5d15-multus-daemon-config\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.754841 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754705    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8161b19f-1a15-4f7d-9232-a9248995abf6-kubelet-config\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:56:12.754841 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754742    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-host-slash\") pod \"iptables-alerter-5zprr\" (UID: \"751f0932-c6b4-4ee6-a4bf-f096e7d27e62\") " pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:56:12.754841 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754791    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhnp\" (UniqueName: \"kubernetes.io/projected/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-kube-api-access-xxhnp\") pod \"iptables-alerter-5zprr\" (UID: \"751f0932-c6b4-4ee6-a4bf-f096e7d27e62\") " pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:56:12.754841 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754828    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:12.755078 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754858    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/569eaf0a-207c-4686-ab02-82e91b45d285-host\") pod \"node-ca-ddszj\" (UID: \"569eaf0a-207c-4686-ab02-82e91b45d285\") " pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:56:12.755078 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754895    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysctl-conf\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.755078 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754913    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-k8s-cni-cncf-io\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.755078 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754955    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-system-cni-dir\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:56:12.755078 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.754989    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-sys-fs\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.755078 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755018    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p99w5\" (UniqueName: \"kubernetes.io/projected/f2762d18-8915-4499-b4e8-4927bde39164-kube-api-access-p99w5\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:12.755078 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755034    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-kubelet\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.755078 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755071    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-slash\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.755405 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755105    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:12.755405 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755125    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-hostroot\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.755405 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755143    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b7058574-e988-4cc5-8a0c-fa961b9080c1-konnectivity-ca\") pod \"konnectivity-agent-k6kl8\" (UID: \"b7058574-e988-4cc5-8a0c-fa961b9080c1\") " pod="kube-system/konnectivity-agent-k6kl8"
Nov 02 11:56:12.755405 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755201    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cni-binary-copy\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:56:12.755405 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755223    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-node-log\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.755405 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755296    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjrdq\" (UniqueName: \"kubernetes.io/projected/96b40a95-1bfc-4388-b366-f5464623ddb5-kube-api-access-vjrdq\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:12.755405 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755360    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3199abe8-4925-4fba-a291-860f9b7394b9-tmp\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.755405 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755377    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-etc-kubernetes\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.755405 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755391    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.755692 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755413    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlr4l\" (UniqueName: \"kubernetes.io/projected/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb-kube-api-access-qlr4l\") pod \"ingress-canary-dc2vq\" (UID: \"037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb\") " pod="openshift-ingress-canary/ingress-canary-dc2vq"
Nov 02 11:56:12.755692 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755415    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cni-binary-copy\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:56:12.755692 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755432    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-run-netns\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.755692 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755499    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3199abe8-4925-4fba-a291-860f9b7394b9-tmp\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.755692 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755534    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-systemd\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.755692 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755554    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-var-lib-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.755692 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755581    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-lib-modules\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.755692 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755627    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmw6l\" (UniqueName: \"kubernetes.io/projected/3199abe8-4925-4fba-a291-860f9b7394b9-kube-api-access-pmw6l\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.755692 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755649    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-cni-multus\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.755692 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755664    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:56:12.755948 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755698    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-config-volume\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:12.755948 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755732    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-cnibin\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.755948 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755773    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:56:12.755948 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755792    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snxcb\" (UniqueName: \"kubernetes.io/projected/569eaf0a-207c-4686-ab02-82e91b45d285-kube-api-access-snxcb\") pod \"node-ca-ddszj\" (UID: \"569eaf0a-207c-4686-ab02-82e91b45d285\") " pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:56:12.755948 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755858    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.755948 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755915    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-modprobe-d\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.756125 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755950    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-run\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.756125 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.755987    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-system-cni-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.756125 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756022    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-cni-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.756125 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756051    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-kubelet\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.756125 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756081    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8161b19f-1a15-4f7d-9232-a9248995abf6-dbus\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:56:12.756125 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756106    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-ovn\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.756281 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756138    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-kubernetes\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.756281 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756159    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:56:12.756281 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756177    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhsj8\" (UniqueName: \"kubernetes.io/projected/5592809c-08dd-46c1-be01-d1da5d6d54b7-kube-api-access-hhsj8\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.756281 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756230    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-env-overrides\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.756427 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756296    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdv8n\" (UniqueName: \"kubernetes.io/projected/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-kube-api-access-xdv8n\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.756459 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756415    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6e44ac17-b852-412a-bc80-5c3d3973d7ed-signing-cabundle\") pod \"service-ca-5fffc97bdf-gffn9\" (UID: \"6e44ac17-b852-412a-bc80-5c3d3973d7ed\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gffn9"
Nov 02 11:56:12.756490 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756462    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk27s\" (UniqueName: \"kubernetes.io/projected/6e44ac17-b852-412a-bc80-5c3d3973d7ed-kube-api-access-zk27s\") pod \"service-ca-5fffc97bdf-gffn9\" (UID: \"6e44ac17-b852-412a-bc80-5c3d3973d7ed\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gffn9"
Nov 02 11:56:12.756531 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756509    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3199abe8-4925-4fba-a291-860f9b7394b9-etc-tuned\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.756561 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756542    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/836a65d8-536a-4183-9f71-4761970d5d15-cni-binary-copy\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.756589 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756562    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cnibin\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:56:12.756589 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756577    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-socket-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.756589 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756576    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3199abe8-4925-4fba-a291-860f9b7394b9-etc-tuned\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.756712 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756693    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2762d18-8915-4499-b4e8-4927bde39164-data-volume\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:12.756742 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756723    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/836a65d8-536a-4183-9f71-4761970d5d15-cni-binary-copy\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.756742 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756729    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-service-ca-bundle\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:12.756794 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756756    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f7641b1-828f-46e8-b276-293aa6ab5615-hosts-file\") pod \"node-resolver-b4wmw\" (UID: \"0f7641b1-828f-46e8-b276-293aa6ab5615\") " pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:56:12.756794 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756780    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-cni-bin\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.756852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756790    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2762d18-8915-4499-b4e8-4927bde39164-data-volume\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:12.756852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756808    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:56:12.756852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756841    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb-cert\") pod \"ingress-canary-dc2vq\" (UID: \"037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb\") " pod="openshift-ingress-canary/ingress-canary-dc2vq"
Nov 02 11:56:12.756950 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756896    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:12.756950 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756936    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovnkube-config\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.757043 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.756965    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovn-node-metrics-cert\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.843020 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.842938    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 11:56:12.843020 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.842945    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 11:56:12.843020 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.842961    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 11:56:12.843020 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.842967    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 11:56:12.843020 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.842959    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 11:56:12.843380 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.843361    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 11:56:12.843380 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.843377    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 11:56:12.843505 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.843414    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 11:56:12.843667 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.843647    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 11:56:12.843667 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.843663    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 11:56:12.843792 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.843666    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 11:56:12.843792 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.843692    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 11:56:12.843792 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.843703    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 11:56:12.843792 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.843709    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 11:56:12.843792 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.843716    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 11:56:12.843792 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.843695    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 11:56:12.843792 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.843729    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 11:56:12.843792 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.843742    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 11:56:12.843792 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.843732    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 11:56:12.843792 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.843719    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 11:56:12.843792 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.843755    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 11:56:12.843792 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.843790    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 11:56:12.844228 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.844147    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wdxfq\""
Nov 02 11:56:12.844228 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.844191    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t47mb\" (UniqueName: \"kubernetes.io/projected/836a65d8-536a-4183-9f71-4761970d5d15-kube-api-access-t47mb\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.844310 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.844291    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 11:56:12.844384 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.844323    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 11:56:12.844384 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.844355    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 11:56:12.844384 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.844376    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 11:56:12.844384 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.844380    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 11:56:12.844570 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.844558    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 11:56:12.844721 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.844708    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:56:12.844794 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.844785    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bjwgn\""
Nov 02 11:56:12.844940 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.844924    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmw6l\" (UniqueName: \"kubernetes.io/projected/3199abe8-4925-4fba-a291-860f9b7394b9-kube-api-access-pmw6l\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.845627 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.845613    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b7058574-e988-4cc5-8a0c-fa961b9080c1-konnectivity-ca\") pod \"konnectivity-agent-k6kl8\" (UID: \"b7058574-e988-4cc5-8a0c-fa961b9080c1\") " pod="kube-system/konnectivity-agent-k6kl8"
Nov 02 11:56:12.845807 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.845792    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 11:56:12.845948 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.845936    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-config-volume\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:12.847007 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.846990    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-service-ca-bundle\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:12.847055 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.847014    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-env-overrides\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.847130 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.847116    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovnkube-config\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.847166 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.847134    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/918aeea0-5975-4be5-a2f4-530e6c38f1ad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:56:12.847166 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.847120    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb-cert\") pod \"ingress-canary-dc2vq\" (UID: \"037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb\") " pod="openshift-ingress-canary/ingress-canary-dc2vq"
Nov 02 11:56:12.849361 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.849345    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 11:56:12.849404 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.849359    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 11:56:12.849626 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.849613    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 11:56:12.850126 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.850112    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlr4l\" (UniqueName: \"kubernetes.io/projected/037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb-kube-api-access-qlr4l\") pod \"ingress-canary-dc2vq\" (UID: \"037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb\") " pod="openshift-ingress-canary/ingress-canary-dc2vq"
Nov 02 11:56:12.850173 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.850122    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjlf7\" (UniqueName: \"kubernetes.io/projected/0f7641b1-828f-46e8-b276-293aa6ab5615-kube-api-access-rjlf7\") pod \"node-resolver-b4wmw\" (UID: \"0f7641b1-828f-46e8-b276-293aa6ab5615\") " pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:56:12.850173 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.850134    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxhnp\" (UniqueName: \"kubernetes.io/projected/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-kube-api-access-xxhnp\") pod \"iptables-alerter-5zprr\" (UID: \"751f0932-c6b4-4ee6-a4bf-f096e7d27e62\") " pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:56:12.850173 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.850143    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjrdq\" (UniqueName: \"kubernetes.io/projected/96b40a95-1bfc-4388-b366-f5464623ddb5-kube-api-access-vjrdq\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:12.850459 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.850443    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbbm7\" (UniqueName: \"kubernetes.io/projected/918aeea0-5975-4be5-a2f4-530e6c38f1ad-kube-api-access-tbbm7\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:56:12.850550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.850531    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85sft\" (UniqueName: \"kubernetes.io/projected/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-kube-api-access-85sft\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:12.850591 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.850568    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lftk\" (UniqueName: \"kubernetes.io/projected/cbfec7e2-d13a-483b-8583-5f19481690c1-kube-api-access-5lftk\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:56:12.850760 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.850746    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p99w5\" (UniqueName: \"kubernetes.io/projected/f2762d18-8915-4499-b4e8-4927bde39164-kube-api-access-p99w5\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:12.854077 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.854061    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lql6t\" (UniqueName: \"kubernetes.io/projected/7c0e97ec-40c1-4554-8966-f2267a4b287c-kube-api-access-lql6t\") pod \"migrator-66df5d496b-fgj5q\" (UID: \"7c0e97ec-40c1-4554-8966-f2267a4b287c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fgj5q"
Nov 02 11:56:12.858188 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858169    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-device-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.858256 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858194    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysctl-d\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.858256 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858220    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-conf-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.858313 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858253    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8161b19f-1a15-4f7d-9232-a9248995abf6-kubelet-config\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:56:12.858313 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858270    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-device-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.858313 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858277    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-host-slash\") pod \"iptables-alerter-5zprr\" (UID: \"751f0932-c6b4-4ee6-a4bf-f096e7d27e62\") " pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:56:12.858313 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858295    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:12.858313 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858309    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/569eaf0a-207c-4686-ab02-82e91b45d285-host\") pod \"node-ca-ddszj\" (UID: \"569eaf0a-207c-4686-ab02-82e91b45d285\") " pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:56:12.858498 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858333    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysctl-conf\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.858498 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858343    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-conf-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.858498 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858355    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-k8s-cni-cncf-io\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.858498 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858373    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-host-slash\") pod \"iptables-alerter-5zprr\" (UID: \"751f0932-c6b4-4ee6-a4bf-f096e7d27e62\") " pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:56:12.858498 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858391    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-system-cni-dir\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:56:12.858498 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858412    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-sys-fs\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.858498 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858426    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysctl-conf\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.858498 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:12.858379    4464 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 11:56:12.858498 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858377    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/569eaf0a-207c-4686-ab02-82e91b45d285-host\") pod \"node-ca-ddszj\" (UID: \"569eaf0a-207c-4686-ab02-82e91b45d285\") " pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:56:12.858498 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858452    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-kubelet\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.858498 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858375    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysctl-d\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.858498 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858427    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-kubelet\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.858498 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858345    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8161b19f-1a15-4f7d-9232-a9248995abf6-kubelet-config\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:56:12.858498 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858474    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-system-cni-dir\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:56:12.858498 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858481    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-sys-fs\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.858498 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858492    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-slash\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:12.858518    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm podName:f2762d18-8915-4499-b4e8-4927bde39164 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:13.358501339 +0000 UTC m=+1.922209739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t692p" (UID: "f2762d18-8915-4499-b4e8-4927bde39164") : configmap "kube-rbac-proxy" not found
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858532    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858393    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-k8s-cni-cncf-io\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858551    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-hostroot\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858519    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-slash\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858584    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-node-log\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858602    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-etc-kubernetes\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:12.858609    4464 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858615    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858635    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-run-netns\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:12.858647    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls podName:19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:13.358632956 +0000 UTC m=+1.922341342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls") pod "dns-default-znfl5" (UID: "19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958") : secret "dns-default-metrics-tls" not found
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858661    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-etc-kubernetes\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858667    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-run-netns\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858671    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-systemd\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858663    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-node-log\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858685    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858696    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-var-lib-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858704    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-systemd\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858712    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-lib-modules\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858727    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-cni-multus\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858678    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-hostroot\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858760    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-var-lib-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858765    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-cni-multus\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858776    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858792    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-cnibin\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858809    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:12.858830    4464 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858842    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-lib-modules\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858853    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-cnibin\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858864    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858831    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:12.858885    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs podName:cbfec7e2-d13a-483b-8583-5f19481690c1 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:13.358867357 +0000 UTC m=+1.922575758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs") pod "network-metrics-daemon-bqwff" (UID: "cbfec7e2-d13a-483b-8583-5f19481690c1") : secret "metrics-daemon-secret" not found
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858886    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858908    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-modprobe-d\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858922    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-run\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858935    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-system-cni-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858950    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-cni-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858961    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-run\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.858963    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-kubelet\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859003    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-system-cni-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859007    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8161b19f-1a15-4f7d-9232-a9248995abf6-dbus\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859005    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-modprobe-d\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859031    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-cni-dir\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859036    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-ovn\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859040    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-kubelet\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859050    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8161b19f-1a15-4f7d-9232-a9248995abf6-dbus\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859063    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-kubernetes\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859062    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-run-ovn\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859082    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6e44ac17-b852-412a-bc80-5c3d3973d7ed-signing-cabundle\") pod \"service-ca-5fffc97bdf-gffn9\" (UID: \"6e44ac17-b852-412a-bc80-5c3d3973d7ed\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gffn9"
Nov 02 11:56:12.859084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859109    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk27s\" (UniqueName: \"kubernetes.io/projected/6e44ac17-b852-412a-bc80-5c3d3973d7ed-kube-api-access-zk27s\") pod \"service-ca-5fffc97bdf-gffn9\" (UID: \"6e44ac17-b852-412a-bc80-5c3d3973d7ed\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gffn9"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859135    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cnibin\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859110    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-kubernetes\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859158    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-socket-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859185    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f7641b1-828f-46e8-b276-293aa6ab5615-hosts-file\") pod \"node-resolver-b4wmw\" (UID: \"0f7641b1-828f-46e8-b276-293aa6ab5615\") " pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859188    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/918aeea0-5975-4be5-a2f4-530e6c38f1ad-cnibin\") pod \"multus-additional-cni-plugins-4gjgd\" (UID: \"918aeea0-5975-4be5-a2f4-530e6c38f1ad\") " pod="openshift-multus/multus-additional-cni-plugins-4gjgd"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859205    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-cni-bin\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859227    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-socket-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859232    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859235    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f7641b1-828f-46e8-b276-293aa6ab5615-hosts-file\") pod \"node-resolver-b4wmw\" (UID: \"0f7641b1-828f-46e8-b276-293aa6ab5615\") " pod="openshift-dns/node-resolver-b4wmw"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859261    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-host\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859288    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-registration-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859263    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-var-lib-cni-bin\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:12.859299    4464 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859319    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859307    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-host\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859347    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-registration-dir\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:12.859353    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls podName:f2762d18-8915-4499-b4e8-4927bde39164 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:13.359339669 +0000 UTC m=+1.923048070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t692p" (UID: "f2762d18-8915-4499-b4e8-4927bde39164") : secret "insights-runtime-extractor-tls" not found
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859369    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6e44ac17-b852-412a-bc80-5c3d3973d7ed-signing-key\") pod \"service-ca-5fffc97bdf-gffn9\" (UID: \"6e44ac17-b852-412a-bc80-5c3d3973d7ed\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gffn9"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859376    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859384    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-sys\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859406    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-log-socket\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859428    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-cni-netd\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859448    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-log-socket\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859450    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-systemd\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859447    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-sys\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859476    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859516    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5592809c-08dd-46c1-be01-d1da5d6d54b7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859478    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-cni-netd\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859527    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-systemd-units\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859478    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-systemd\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859556    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-run-ovn-kubernetes\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859565    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-systemd-units\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859582    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-cni-bin\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859589    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-run-ovn-kubernetes\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859609    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-multus-certs\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859620    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-host-cni-bin\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859634    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2762d18-8915-4499-b4e8-4927bde39164-crio-socket\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859882    4464 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859924    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-etc-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859950    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-multus-certs\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.859969    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-var-lib-kubelet\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.860021    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2762d18-8915-4499-b4e8-4927bde39164-crio-socket\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.860034    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-etc-openvswitch\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.860035    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-netns\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.860079    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysconfig\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.860086    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-var-lib-kubelet\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.860045    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6e44ac17-b852-412a-bc80-5c3d3973d7ed-signing-cabundle\") pod \"service-ca-5fffc97bdf-gffn9\" (UID: \"6e44ac17-b852-412a-bc80-5c3d3973d7ed\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gffn9"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.860122    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-socket-dir-parent\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.860170    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-multus-socket-dir-parent\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.860178    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/836a65d8-536a-4183-9f71-4761970d5d15-host-run-netns\") pod \"multus-7wp5n\" (UID: \"836a65d8-536a-4183-9f71-4761970d5d15\") " pod="openshift-multus/multus-7wp5n"
Nov 02 11:56:12.861084 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.860224    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3199abe8-4925-4fba-a291-860f9b7394b9-etc-sysconfig\") pod \"tuned-wdmr4\" (UID: \"3199abe8-4925-4fba-a291-860f9b7394b9\") " pod="openshift-cluster-node-tuning-operator/tuned-wdmr4"
Nov 02 11:56:12.864327 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.864307    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6e44ac17-b852-412a-bc80-5c3d3973d7ed-signing-key\") pod \"service-ca-5fffc97bdf-gffn9\" (UID: \"6e44ac17-b852-412a-bc80-5c3d3973d7ed\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gffn9"
Nov 02 11:56:12.864674 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.864658    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 11:56:12.864774 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.864757    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 11:56:12.868053 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.868036    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovn-node-metrics-cert\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.870268 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.870253    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 11:56:12.870565 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.870525    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 11:56:12.870565 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.870535    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4pvvq\""
Nov 02 11:56:12.870565 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.870537    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-lk8z8\""
Nov 02 11:56:12.870565 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.870555    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 11:56:12.870565 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.870530    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 11:56:12.870770 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.870566    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 11:56:12.870770 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.870631    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhsj8\" (UniqueName: \"kubernetes.io/projected/5592809c-08dd-46c1-be01-d1da5d6d54b7-kube-api-access-hhsj8\") pod \"aws-ebs-csi-driver-node-7dnp7\" (UID: \"5592809c-08dd-46c1-be01-d1da5d6d54b7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7dnp7"
Nov 02 11:56:12.870891 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.870808    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-g4drg\""
Nov 02 11:56:12.870891 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.870808    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 11:56:12.870891 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.870823    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qcv25\""
Nov 02 11:56:12.870891 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.870827    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gpdd6\""
Nov 02 11:56:12.870891 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.870850    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vv2bx\""
Nov 02 11:56:12.870891 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.870857    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-n2k42\""
Nov 02 11:56:12.870891 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.870859    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4wp8v\""
Nov 02 11:56:12.870891 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.870878    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hqqr9\""
Nov 02 11:56:12.874494 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.874480    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/569eaf0a-207c-4686-ab02-82e91b45d285-serviceca\") pod \"node-ca-ddszj\" (UID: \"569eaf0a-207c-4686-ab02-82e91b45d285\") " pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:56:12.875519 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.875482    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 11:56:12.875519 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.875493    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q864t\""
Nov 02 11:56:12.875519 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.875502    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hc67r\""
Nov 02 11:56:12.881123 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.881102    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snxcb\" (UniqueName: \"kubernetes.io/projected/569eaf0a-207c-4686-ab02-82e91b45d285-kube-api-access-snxcb\") pod \"node-ca-ddszj\" (UID: \"569eaf0a-207c-4686-ab02-82e91b45d285\") " pod="openshift-image-registry/node-ca-ddszj"
Nov 02 11:56:12.881201 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.881145    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdv8n\" (UniqueName: \"kubernetes.io/projected/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-kube-api-access-xdv8n\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.881201 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.881176    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srjx5\" (UniqueName: \"kubernetes.io/projected/2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab-kube-api-access-srjx5\") pod \"network-check-target-x4hmk\" (UID: \"2d717e6f-b6a7-48e7-a04a-c90a78b0e5ab\") " pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:56:12.884138 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.884121    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b7058574-e988-4cc5-8a0c-fa961b9080c1-agent-certs\") pod \"konnectivity-agent-k6kl8\" (UID: \"b7058574-e988-4cc5-8a0c-fa961b9080c1\") " pod="kube-system/konnectivity-agent-k6kl8"
Nov 02 11:56:12.892103 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.892088    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mvmxd\""
Nov 02 11:56:12.892183 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.892108    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 11:56:12.892360 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.892346    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2rrmx\""
Nov 02 11:56:12.892404 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.892362    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x7vnq\""
Nov 02 11:56:12.892449 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.892420    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hx94g\""
Nov 02 11:56:12.893513 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.893499    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/751f0932-c6b4-4ee6-a4bf-f096e7d27e62-iptables-alerter-script\") pod \"iptables-alerter-5zprr\" (UID: \"751f0932-c6b4-4ee6-a4bf-f096e7d27e62\") " pod="openshift-network-operator/iptables-alerter-5zprr"
Nov 02 11:56:12.894251 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.894238    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 11:56:12.894843 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.894832    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 11:56:12.895065 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.895051    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 11:56:12.897118 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.897100    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8161b19f-1a15-4f7d-9232-a9248995abf6-original-pull-secret\") pod \"global-pull-secret-syncer-sw29m\" (UID: \"8161b19f-1a15-4f7d-9232-a9248995abf6\") " pod="kube-system/global-pull-secret-syncer-sw29m"
Nov 02 11:56:12.905338 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.905319    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-trusted-ca-bundle\") pod \"periodic-gathering-vfccn-qnzjq\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") " pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:56:12.905414 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.905328    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d66d28c-8f94-4e4f-a07a-10b188c9bc6a-ovnkube-script-lib\") pod \"ovnkube-node-w62gt\" (UID: \"9d66d28c-8f94-4e4f-a07a-10b188c9bc6a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:12.920886 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.920864    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk27s\" (UniqueName: \"kubernetes.io/projected/6e44ac17-b852-412a-bc80-5c3d3973d7ed-kube-api-access-zk27s\") pod \"service-ca-5fffc97bdf-gffn9\" (UID: \"6e44ac17-b852-412a-bc80-5c3d3973d7ed\") " pod="openshift-service-ca/service-ca-5fffc97bdf-gffn9"
Nov 02 11:56:12.996855 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:12.996836    4464 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 11:56:13.038813 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:13.038791    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-gffn9"
Nov 02 11:56:13.049109 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:13.049086    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:56:13.364819 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:13.364743    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:56:13.364819 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:13.364787    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:13.364819 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:13.364818    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:13.365046 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:13.364901    4464 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 11:56:13.365046 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:13.364933    4464 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 11:56:13.365046 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:13.364954    4464 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 11:56:13.365046 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:13.364964    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs podName:cbfec7e2-d13a-483b-8583-5f19481690c1 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:14.364949938 +0000 UTC m=+2.928658325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs") pod "network-metrics-daemon-bqwff" (UID: "cbfec7e2-d13a-483b-8583-5f19481690c1") : secret "metrics-daemon-secret" not found
Nov 02 11:56:13.365046 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:13.364905    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:13.365046 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:13.364905    4464 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 11:56:13.365046 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:13.365002    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm podName:f2762d18-8915-4499-b4e8-4927bde39164 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:14.36497113 +0000 UTC m=+2.928679538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t692p" (UID: "f2762d18-8915-4499-b4e8-4927bde39164") : configmap "kube-rbac-proxy" not found
Nov 02 11:56:13.365046 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:13.365020    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls podName:19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:14.365013533 +0000 UTC m=+2.928721920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls") pod "dns-default-znfl5" (UID: "19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958") : secret "dns-default-metrics-tls" not found
Nov 02 11:56:13.365046 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:13.365039    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls podName:f2762d18-8915-4499-b4e8-4927bde39164 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:14.365030089 +0000 UTC m=+2.928738480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t692p" (UID: "f2762d18-8915-4499-b4e8-4927bde39164") : secret "insights-runtime-extractor-tls" not found
Nov 02 11:56:13.435232 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:13.435205    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-gffn9"]
Nov 02 11:56:13.438276 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:13.438242    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e44ac17_b852_412a_bc80_5c3d3973d7ed.slice/crio-e66c0d03713d43940c9b1241fb5d6c048c10cadb52d0dbc806f94e07124a9182 WatchSource:0}: Error finding container e66c0d03713d43940c9b1241fb5d6c048c10cadb52d0dbc806f94e07124a9182: Status 404 returned error can't find the container with id e66c0d03713d43940c9b1241fb5d6c048c10cadb52d0dbc806f94e07124a9182
Nov 02 11:56:13.612540 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:13.612514    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-gffn9" event={"ID":"6e44ac17-b852-412a-bc80-5c3d3973d7ed","Type":"ContainerStarted","Data":"e66c0d03713d43940c9b1241fb5d6c048c10cadb52d0dbc806f94e07124a9182"}
Nov 02 11:56:14.372445 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:14.372402    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:14.372620 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:14.372460    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:14.372620 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:14.372522    4464 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 11:56:14.372620 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:14.372549    4464 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 11:56:14.372620 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:14.372573    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:56:14.372620 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:14.372589    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm podName:f2762d18-8915-4499-b4e8-4927bde39164 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:16.372572491 +0000 UTC m=+4.936280891 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t692p" (UID: "f2762d18-8915-4499-b4e8-4927bde39164") : configmap "kube-rbac-proxy" not found
Nov 02 11:56:14.372620 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:14.372619    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls podName:19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:16.372606011 +0000 UTC m=+4.936314410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls") pod "dns-default-znfl5" (UID: "19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958") : secret "dns-default-metrics-tls" not found
Nov 02 11:56:14.372937 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:14.372643    4464 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 11:56:14.372937 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:14.372670    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:14.372937 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:14.372680    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs podName:cbfec7e2-d13a-483b-8583-5f19481690c1 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:16.37266874 +0000 UTC m=+4.936377133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs") pod "network-metrics-daemon-bqwff" (UID: "cbfec7e2-d13a-483b-8583-5f19481690c1") : secret "metrics-daemon-secret" not found
Nov 02 11:56:14.372937 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:14.372734    4464 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 11:56:14.372937 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:14.372790    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls podName:f2762d18-8915-4499-b4e8-4927bde39164 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:16.372779341 +0000 UTC m=+4.936487733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t692p" (UID: "f2762d18-8915-4499-b4e8-4927bde39164") : secret "insights-runtime-extractor-tls" not found
Nov 02 11:56:15.620453 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:15.620418    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fgj5q" event={"ID":"7c0e97ec-40c1-4554-8966-f2267a4b287c","Type":"ContainerStarted","Data":"d0eddbdd23f48880dfa9503af2013f242c7d60757eef4629bc66eb46a3bbf3ae"}
Nov 02 11:56:16.390186 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:16.390098    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:16.390186 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:16.390154    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:16.390395 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:16.390187    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:56:16.390395 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:16.390228    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:16.390395 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:16.390315    4464 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 11:56:16.390395 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:16.390333    4464 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 11:56:16.390395 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:16.390376    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls podName:19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:20.390361564 +0000 UTC m=+8.954069951 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls") pod "dns-default-znfl5" (UID: "19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958") : secret "dns-default-metrics-tls" not found
Nov 02 11:56:16.390618 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:16.390402    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs podName:cbfec7e2-d13a-483b-8583-5f19481690c1 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:20.390385224 +0000 UTC m=+8.954093628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs") pod "network-metrics-daemon-bqwff" (UID: "cbfec7e2-d13a-483b-8583-5f19481690c1") : secret "metrics-daemon-secret" not found
Nov 02 11:56:16.390618 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:16.390340    4464 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 11:56:16.390618 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:16.390447    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls podName:f2762d18-8915-4499-b4e8-4927bde39164 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:20.3904368 +0000 UTC m=+8.954145217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t692p" (UID: "f2762d18-8915-4499-b4e8-4927bde39164") : secret "insights-runtime-extractor-tls" not found
Nov 02 11:56:16.390715 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:16.390692    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2762d18-8915-4499-b4e8-4927bde39164-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:16.624416 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:16.624380    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-gffn9" event={"ID":"6e44ac17-b852-412a-bc80-5c3d3973d7ed","Type":"ContainerStarted","Data":"84525e1b2bca475c5fea361f524a25ead40c2534975e86b3b1b19fd6605ea6da"}
Nov 02 11:56:16.625914 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:16.625889    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vfccn-qnzjq" event={"ID":"96b40a95-1bfc-4388-b366-f5464623ddb5","Type":"ContainerStarted","Data":"491d92ca92d6f67a078b6d7d3338af189b202ee9658f50af9acd79c2813ded58"}
Nov 02 11:56:16.741404 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:16.741350    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-gffn9" podStartSLOduration=3.156456306 podStartE2EDuration="5.74133988s" totalImagesPullingTime="2.584883574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:16.741018001 +0000 UTC m=+5.304726407" watchObservedRunningTime="2026-11-02 11:56:16.74133988 +0000 UTC m=+5.305048286"
Nov 02 11:56:18.877269 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:18.877239    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b4wmw_0f7641b1-828f-46e8-b276-293aa6ab5615/dns-node-resolver/0.log"
Nov 02 11:56:19.102154 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:19.102126    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:19.779578 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:19.779550    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:19.779758 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:19.779747    4464 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 11:56:19.871725 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:19.871697    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ddszj_569eaf0a-207c-4686-ab02-82e91b45d285/node-ca/0.log"
Nov 02 11:56:20.419295 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:20.419261    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:20.419686 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:20.419307    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:20.419686 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:20.419333    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:56:20.419686 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:20.419412    4464 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 11:56:20.419686 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:20.419475    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls podName:19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:28.419462445 +0000 UTC m=+16.983170832 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls") pod "dns-default-znfl5" (UID: "19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958") : secret "dns-default-metrics-tls" not found
Nov 02 11:56:20.419686 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:20.419414    4464 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 11:56:20.419686 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:20.419568    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls podName:f2762d18-8915-4499-b4e8-4927bde39164 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:28.419555771 +0000 UTC m=+16.983264181 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t692p" (UID: "f2762d18-8915-4499-b4e8-4927bde39164") : secret "insights-runtime-extractor-tls" not found
Nov 02 11:56:20.421636 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:20.421611    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbfec7e2-d13a-483b-8583-5f19481690c1-metrics-certs\") pod \"network-metrics-daemon-bqwff\" (UID: \"cbfec7e2-d13a-483b-8583-5f19481690c1\") " pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:56:20.488251 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:20.488226    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bqwff"
Nov 02 11:56:20.672178 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:20.672103    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dc2vq_037bfa58-bb7c-4ec4-8cc3-a1c7e72afbfb/serve-healthcheck-canary/0.log"
Nov 02 11:56:20.852729 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:20.852700    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bqwff"]
Nov 02 11:56:20.856773 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:20.856745    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbfec7e2_d13a_483b_8583_5f19481690c1.slice/crio-5e2a758ffb901460e1e7ed964e0748ec651d34f01649ce70a64b17bcdd790e3d WatchSource:0}: Error finding container 5e2a758ffb901460e1e7ed964e0748ec651d34f01649ce70a64b17bcdd790e3d: Status 404 returned error can't find the container with id 5e2a758ffb901460e1e7ed964e0748ec651d34f01649ce70a64b17bcdd790e3d
Nov 02 11:56:21.072142 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:21.072113    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-fgj5q_7c0e97ec-40c1-4554-8966-f2267a4b287c/migrator/0.log"
Nov 02 11:56:21.639374 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:21.639342    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bqwff" event={"ID":"cbfec7e2-d13a-483b-8583-5f19481690c1","Type":"ContainerStarted","Data":"5e2a758ffb901460e1e7ed964e0748ec651d34f01649ce70a64b17bcdd790e3d"}
Nov 02 11:56:22.643454 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:22.643384    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bqwff" event={"ID":"cbfec7e2-d13a-483b-8583-5f19481690c1","Type":"ContainerStarted","Data":"2d3eb24c8f37c346d07aab18088390811f3b0ff98fa9d6431fd33d4e8eed02b9"}
Nov 02 11:56:22.643784 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:22.643464    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bqwff" event={"ID":"cbfec7e2-d13a-483b-8583-5f19481690c1","Type":"ContainerStarted","Data":"c5cee8ccce39b05ae572481e2f56fe6acc492448c334468e8305ee37d3f3bf4a"}
Nov 02 11:56:24.590043 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:24.590010    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w62gt"
Nov 02 11:56:28.471806 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:28.471763    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:28.472247 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:28.471821    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:28.474050 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:28.474020    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2762d18-8915-4499-b4e8-4927bde39164-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t692p\" (UID: \"f2762d18-8915-4499-b4e8-4927bde39164\") " pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:28.474154 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:28.474057    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958-metrics-tls\") pod \"dns-default-znfl5\" (UID: \"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958\") " pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:28.594243 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:28.594213    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:28.595298 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:28.595280    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t692p"
Nov 02 11:56:29.019372 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:29.019349    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-znfl5"]
Nov 02 11:56:29.021806 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:29.021779    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19fc04ac_7a12_4feb_ad4a_a6b2fe6f8958.slice/crio-438d3adbbac4713f2ca4f0be265bcdc6f75080512f96a4a3a6d47a8e8ab525d3 WatchSource:0}: Error finding container 438d3adbbac4713f2ca4f0be265bcdc6f75080512f96a4a3a6d47a8e8ab525d3: Status 404 returned error can't find the container with id 438d3adbbac4713f2ca4f0be265bcdc6f75080512f96a4a3a6d47a8e8ab525d3
Nov 02 11:56:29.059471 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:29.059451    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2762d18_8915_4499_b4e8_4927bde39164.slice/crio-0773b2c12208fea97286447da406c04dbe0769269a5b8eeeeebc3f100c5f7352 WatchSource:0}: Error finding container 0773b2c12208fea97286447da406c04dbe0769269a5b8eeeeebc3f100c5f7352: Status 404 returned error can't find the container with id 0773b2c12208fea97286447da406c04dbe0769269a5b8eeeeebc3f100c5f7352
Nov 02 11:56:29.070113 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:29.070094    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t692p"]
Nov 02 11:56:29.668184 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:29.668148    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t692p" event={"ID":"f2762d18-8915-4499-b4e8-4927bde39164","Type":"ContainerStarted","Data":"4d6f0bd55271765b9576d08357806894184a73d8bb90061653a88c6d2873ba85"}
Nov 02 11:56:29.668580 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:29.668230    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t692p" event={"ID":"f2762d18-8915-4499-b4e8-4927bde39164","Type":"ContainerStarted","Data":"0773b2c12208fea97286447da406c04dbe0769269a5b8eeeeebc3f100c5f7352"}
Nov 02 11:56:29.669307 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:29.669278    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-znfl5" event={"ID":"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958","Type":"ContainerStarted","Data":"438d3adbbac4713f2ca4f0be265bcdc6f75080512f96a4a3a6d47a8e8ab525d3"}
Nov 02 11:56:30.674455 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:30.674376    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t692p" event={"ID":"f2762d18-8915-4499-b4e8-4927bde39164","Type":"ContainerStarted","Data":"9abf28ceafffb88368bf66b6259838817e304cdde2e91aa9342cc4fc9be2f971"}
Nov 02 11:56:31.682875 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:31.682836    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-znfl5" event={"ID":"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958","Type":"ContainerStarted","Data":"a3c5df7a61eca48242b0a2260f77ce4aaf5174fd89241fc0b0e63d0de70c1c65"}
Nov 02 11:56:31.683330 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:31.682893    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-znfl5" event={"ID":"19fc04ac-7a12-4feb-ad4a-a6b2fe6f8958","Type":"ContainerStarted","Data":"25a72f995f8f8e4c9b0b35f989a4b77d4c34c23295ef5d2d470234d5a1728869"}
Nov 02 11:56:31.683330 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:31.683071    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:32.687638 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:32.687598    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t692p" event={"ID":"f2762d18-8915-4499-b4e8-4927bde39164","Type":"ContainerStarted","Data":"16fbd49f8c76b5edf4240380613c03a494d60d7016dd9aac71263104cf8775f6"}
Nov 02 11:56:40.768771 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:40.768739    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k6kl8"
Nov 02 11:56:41.689474 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:41.689448    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-znfl5"
Nov 02 11:56:43.615313 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:43.615285    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x4hmk"
Nov 02 11:56:44.799511 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.799477    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ccmmd"]
Nov 02 11:56:44.831325 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.831297    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:44.850049 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.850026    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"]
Nov 02 11:56:44.863278 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.863258    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:44.884405 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.884371    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 11:56:44.884480 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.884464    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 11:56:44.900111 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.900087    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 11:56:44.900182 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.900149    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 11:56:44.935144 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.935124    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 11:56:44.935252 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.935163    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-brq6w\""
Nov 02 11:56:44.935252 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.935214    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 11:56:44.935843 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.935828    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"]
Nov 02 11:56:44.969027 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.969006    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 02 11:56:44.969095 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.969065    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 02 11:56:44.969095 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.969070    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pbmdx\""
Nov 02 11:56:44.973444 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.973423    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-sys\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:44.973496 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.973455    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-node-exporter-textfile\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:44.973496 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.973471    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-metrics-client-ca\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:44.973561 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.973491    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-node-exporter-accelerators-collector-config\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:44.973561 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.973515    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-node-exporter-tls\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:44.973650 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.973628    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bcd1b647-47f8-4785-bfb6-acfa67de94fe-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:44.973709 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.973696    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcd1b647-47f8-4785-bfb6-acfa67de94fe-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:44.973742 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.973720    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-root\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:44.973742 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.973737    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:44.973823 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.973762    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skmrb\" (UniqueName: \"kubernetes.io/projected/bcd1b647-47f8-4785-bfb6-acfa67de94fe-kube-api-access-skmrb\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:44.973891 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.973879    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7flnm\" (UniqueName: \"kubernetes.io/projected/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-kube-api-access-7flnm\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:44.973934 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.973904    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bcd1b647-47f8-4785-bfb6-acfa67de94fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:44.973934 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.973919    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-node-exporter-wtmp\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:44.974070 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.973963    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcd1b647-47f8-4785-bfb6-acfa67de94fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:44.974128 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.974089    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bcd1b647-47f8-4785-bfb6-acfa67de94fe-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:44.986014 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:44.985996    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 02 11:56:45.074381 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.074320    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-node-exporter-textfile\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.074381 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.074348    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-metrics-client-ca\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.074381 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.074366    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-node-exporter-accelerators-collector-config\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.074610 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.074384    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-node-exporter-tls\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.074610 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.074403    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bcd1b647-47f8-4785-bfb6-acfa67de94fe-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:45.074610 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.074426    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcd1b647-47f8-4785-bfb6-acfa67de94fe-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:45.074610 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.074449    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-root\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.074814 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.074755    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-root\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.074814 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.074764    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-node-exporter-textfile\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.074814 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.074796    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.074964 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.074827    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skmrb\" (UniqueName: \"kubernetes.io/projected/bcd1b647-47f8-4785-bfb6-acfa67de94fe-kube-api-access-skmrb\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:45.074964 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.074866    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7flnm\" (UniqueName: \"kubernetes.io/projected/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-kube-api-access-7flnm\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.074964 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.074902    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bcd1b647-47f8-4785-bfb6-acfa67de94fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:45.074964 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.074926    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-node-exporter-wtmp\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.075179 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.074967    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcd1b647-47f8-4785-bfb6-acfa67de94fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:45.075179 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.075030    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bcd1b647-47f8-4785-bfb6-acfa67de94fe-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:45.075179 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.075056    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-metrics-client-ca\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.075179 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.075064    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-sys\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.075179 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.075128    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-node-exporter-accelerators-collector-config\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.075428 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.075217    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-node-exporter-wtmp\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.075500 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.075476    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcd1b647-47f8-4785-bfb6-acfa67de94fe-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:45.075582 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.075547    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-sys\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.075649 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.075573    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bcd1b647-47f8-4785-bfb6-acfa67de94fe-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:45.075804 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.075785    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bcd1b647-47f8-4785-bfb6-acfa67de94fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:45.077144 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.077122    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-node-exporter-tls\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.077264 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.077247    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bcd1b647-47f8-4785-bfb6-acfa67de94fe-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:45.077473 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.077453    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.077791 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.077772    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcd1b647-47f8-4785-bfb6-acfa67de94fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:45.136960 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.136932    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7flnm\" (UniqueName: \"kubernetes.io/projected/9fffe20a-88c8-4196-8b56-5ddf2c71fcaf-kube-api-access-7flnm\") pod \"node-exporter-ccmmd\" (UID: \"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf\") " pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.137129 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.137108    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skmrb\" (UniqueName: \"kubernetes.io/projected/bcd1b647-47f8-4785-bfb6-acfa67de94fe-kube-api-access-skmrb\") pod \"kube-state-metrics-5dc8d7d89b-7ncxv\" (UID: \"bcd1b647-47f8-4785-bfb6-acfa67de94fe\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:45.183620 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.183596    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ccmmd"
Nov 02 11:56:45.220650 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.220630    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"
Nov 02 11:56:45.223410 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:45.223387    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fffe20a_88c8_4196_8b56_5ddf2c71fcaf.slice/crio-16b4dbda43c4133ecda5c67d32650c1b16db8877e6c8f74d61025052b69088f7 WatchSource:0}: Error finding container 16b4dbda43c4133ecda5c67d32650c1b16db8877e6c8f74d61025052b69088f7: Status 404 returned error can't find the container with id 16b4dbda43c4133ecda5c67d32650c1b16db8877e6c8f74d61025052b69088f7
Nov 02 11:56:45.665064 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.664882    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv"]
Nov 02 11:56:45.669365 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:45.669335    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd1b647_47f8_4785_bfb6_acfa67de94fe.slice/crio-8d9c3b8bc76168cbedcbdc95f4b319b5ab681bdff28104c13067b24d2c366e06 WatchSource:0}: Error finding container 8d9c3b8bc76168cbedcbdc95f4b319b5ab681bdff28104c13067b24d2c366e06: Status 404 returned error can't find the container with id 8d9c3b8bc76168cbedcbdc95f4b319b5ab681bdff28104c13067b24d2c366e06
Nov 02 11:56:45.729273 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.729246    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv" event={"ID":"bcd1b647-47f8-4785-bfb6-acfa67de94fe","Type":"ContainerStarted","Data":"8d9c3b8bc76168cbedcbdc95f4b319b5ab681bdff28104c13067b24d2c366e06"}
Nov 02 11:56:45.730382 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:45.730359    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ccmmd" event={"ID":"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf","Type":"ContainerStarted","Data":"16b4dbda43c4133ecda5c67d32650c1b16db8877e6c8f74d61025052b69088f7"}
Nov 02 11:56:47.099753 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.099727    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 11:56:47.108235 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.108211    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.151314 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.151289    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 11:56:47.188764 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.188699    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.188764 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.188740    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.188937 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.188807    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.188990 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.188928    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.188990 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.188957    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.189117 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.188998    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.189117 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.189101    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.189212 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.189137    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-config-out\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.189212 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.189164    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwwl8\" (UniqueName: \"kubernetes.io/projected/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-kube-api-access-wwwl8\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.189295 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.189246    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.189342 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.189291    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.189342 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.189310    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.189451 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.189339    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-web-config\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.195002 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.194232    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 11:56:47.195002 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.194271    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 11:56:47.195002 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.194293    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 11:56:47.195002 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.194303    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 11:56:47.195002 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.194315    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 11:56:47.195002 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.194354    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 11:56:47.195002 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.194432    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bbkkg\""
Nov 02 11:56:47.263265 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.263240    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 11:56:47.263416 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.263307    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 11:56:47.263416 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.263312    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 11:56:47.289890 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.289865    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.290052 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.289894    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.290052 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.289913    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.290052 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.289931    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.290052 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.289949    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.290052 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.289971    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.290295 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.290105    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-config-out\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.290295 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.290148    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwwl8\" (UniqueName: \"kubernetes.io/projected/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-kube-api-access-wwwl8\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.290295 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.290201    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.290295 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.290237    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.290295 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.290260    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.290547 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.290297    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-web-config\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.290547 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.290345    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.290547 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.290353    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.297056 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.297034    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.297555 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.297220    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.297555 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.297367    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.297555 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.297516    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.297555 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.297523    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.297803 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.297680    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.297864 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.297851    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.298950 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.298924    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-config-out\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.299115 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.299094    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.299382 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.299366    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-web-config\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.299484 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.299468    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.348211 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.348186    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"]
Nov 02 11:56:47.357932 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.357905    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwwl8\" (UniqueName: \"kubernetes.io/projected/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-kube-api-access-wwwl8\") pod \"alertmanager-main-0\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.359759 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.359741    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.414038 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.413396    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 11:56:47.414038 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.413547    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 11:56:47.414038 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.413650    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6b4mirq72cc34\""
Nov 02 11:56:47.414038 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.414013    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 11:56:47.414865 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.414843    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 11:56:47.414961 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.414908    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 11:56:47.415039 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.415011    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5fv2t\""
Nov 02 11:56:47.415369 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.415346    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"]
Nov 02 11:56:47.491428 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.491396    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.491523 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.491442    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-metrics-client-ca\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.491561 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.491521    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.491561 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.491551    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.491634 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.491573    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-grpc-tls\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.491681 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.491666    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-tmp\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.491716 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.491692    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5hj2\" (UniqueName: \"kubernetes.io/projected/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-kube-api-access-g5hj2\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.491755 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.491742    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.491797 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.491770    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-thanos-querier-tls\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.535552 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.535530    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:56:47.592545 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.592515    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-tmp\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.592664 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.592574    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5hj2\" (UniqueName: \"kubernetes.io/projected/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-kube-api-access-g5hj2\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.592664 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.592613    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.592664 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.592651    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-thanos-querier-tls\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.592760 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.592691    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.592760 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.592726    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-metrics-client-ca\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.592822 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.592760    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.592822 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.592790    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.592881 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.592818    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-grpc-tls\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.594804 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.594777    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-metrics-client-ca\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.595487 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.595129    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-grpc-tls\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.595487 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.595320    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-thanos-querier-tls\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.595799 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.595587    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-tmp\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.597715 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.597673    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.597828 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.597812    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.597912 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.597820    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.599032 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.598156    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.656529 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.656507    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5hj2\" (UniqueName: \"kubernetes.io/projected/06d0e7da-fbf7-4c6b-b532-9ef01ab60150-kube-api-access-g5hj2\") pod \"thanos-querier-7b4bb8cbdb-fh5rn\" (UID: \"06d0e7da-fbf7-4c6b-b532-9ef01ab60150\") " pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.716635 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.716615    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:47.739411 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.739385    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv" event={"ID":"bcd1b647-47f8-4785-bfb6-acfa67de94fe","Type":"ContainerStarted","Data":"fbc1c9a0ccabb99eed5e6d71b08c81efdbf0c79da1844ed349bd56d42dc96e16"}
Nov 02 11:56:47.739524 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.739442    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv" event={"ID":"bcd1b647-47f8-4785-bfb6-acfa67de94fe","Type":"ContainerStarted","Data":"a5cd0af1f1e4ebcdfa9b10554b13618ec2c62f9c338eed2e20e6dfebc988f84a"}
Nov 02 11:56:47.739524 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.739471    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv" event={"ID":"bcd1b647-47f8-4785-bfb6-acfa67de94fe","Type":"ContainerStarted","Data":"a99bbbe401badedc77ae4110cf8ab82cb700ad63048bb769e3be2047b7a92631"}
Nov 02 11:56:47.741076 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.741045    4464 generic.go:356] "Generic (PLEG): container finished" podID="9fffe20a-88c8-4196-8b56-5ddf2c71fcaf" containerID="c1e847e23db784cc256b69f442c7487ab20d035203d62a7b21e09d890292f0b9" exitCode=0
Nov 02 11:56:47.741194 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:47.741092    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ccmmd" event={"ID":"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf","Type":"ContainerDied","Data":"c1e847e23db784cc256b69f442c7487ab20d035203d62a7b21e09d890292f0b9"}
Nov 02 11:56:48.036248 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:48.036170    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 11:56:48.039423 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:48.039397    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d58c0d_d8a5_4074_a08d_47c32c99a5a7.slice/crio-d027bfbd2c55bf8eca5f91cc6abbdc41708f9e939f4abe14fdf737c5104e5a66 WatchSource:0}: Error finding container d027bfbd2c55bf8eca5f91cc6abbdc41708f9e939f4abe14fdf737c5104e5a66: Status 404 returned error can't find the container with id d027bfbd2c55bf8eca5f91cc6abbdc41708f9e939f4abe14fdf737c5104e5a66
Nov 02 11:56:48.151836 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:48.151800    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"]
Nov 02 11:56:48.154359 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:48.154334    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d0e7da_fbf7_4c6b_b532_9ef01ab60150.slice/crio-79fca1c5f8651d538d167aa092b0bf3bf86be8450d491ed10d0944f82cbe5445 WatchSource:0}: Error finding container 79fca1c5f8651d538d167aa092b0bf3bf86be8450d491ed10d0944f82cbe5445: Status 404 returned error can't find the container with id 79fca1c5f8651d538d167aa092b0bf3bf86be8450d491ed10d0944f82cbe5445
Nov 02 11:56:48.749698 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:48.749662    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ccmmd" event={"ID":"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf","Type":"ContainerStarted","Data":"d94b2c118f72b82103aadd746e535efb57b13c6228de1bfe451d67a89f5d3ce0"}
Nov 02 11:56:48.749950 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:48.749740    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ccmmd" event={"ID":"9fffe20a-88c8-4196-8b56-5ddf2c71fcaf","Type":"ContainerStarted","Data":"733ecb8c576513206a52795fc4a63b2e265f0d7f9f08ae215e8e61dc4d23bffe"}
Nov 02 11:56:48.750900 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:48.750876    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn" event={"ID":"06d0e7da-fbf7-4c6b-b532-9ef01ab60150","Type":"ContainerStarted","Data":"79fca1c5f8651d538d167aa092b0bf3bf86be8450d491ed10d0944f82cbe5445"}
Nov 02 11:56:48.751928 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:48.751897    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7","Type":"ContainerStarted","Data":"d027bfbd2c55bf8eca5f91cc6abbdc41708f9e939f4abe14fdf737c5104e5a66"}
Nov 02 11:56:48.870396 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:48.870342    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ccmmd" podStartSLOduration=3.098077317 podStartE2EDuration="4.870329854s" totalImagesPullingTime="1.693617075s" totalInitContainerRuntime="78.635462ms" isStatefulPod=true podCreationTimestamp="2026-11-02 11:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:48.870031497 +0000 UTC m=+37.433739908" watchObservedRunningTime="2026-11-02 11:56:48.870329854 +0000 UTC m=+37.434038251"
Nov 02 11:56:48.995859 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:48.995808    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7ncxv" podStartSLOduration=3.182319526 podStartE2EDuration="4.995794036s" totalImagesPullingTime="1.81347451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:48.995365192 +0000 UTC m=+37.559073602" watchObservedRunningTime="2026-11-02 11:56:48.995794036 +0000 UTC m=+37.559502485"
Nov 02 11:56:49.510545 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.510519    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rqs2c"]
Nov 02 11:56:49.513210 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.512805    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqs2c"
Nov 02 11:56:49.572031 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.572003    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rqs2c"]
Nov 02 11:56:49.572129 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.572122    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 02 11:56:49.572502 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.572485    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-d8gxw\""
Nov 02 11:56:49.608587 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.608559    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c3b3675-ec09-40d1-a2e9-0a74dcb48286-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rqs2c\" (UID: \"7c3b3675-ec09-40d1-a2e9-0a74dcb48286\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqs2c"
Nov 02 11:56:49.608692 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.608640    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vrnw\" (UniqueName: \"kubernetes.io/projected/7c3b3675-ec09-40d1-a2e9-0a74dcb48286-kube-api-access-5vrnw\") pod \"monitoring-plugin-77f4f76594-rqs2c\" (UID: \"7c3b3675-ec09-40d1-a2e9-0a74dcb48286\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqs2c"
Nov 02 11:56:49.709542 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.709512    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c3b3675-ec09-40d1-a2e9-0a74dcb48286-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rqs2c\" (UID: \"7c3b3675-ec09-40d1-a2e9-0a74dcb48286\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqs2c"
Nov 02 11:56:49.709680 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.709560    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vrnw\" (UniqueName: \"kubernetes.io/projected/7c3b3675-ec09-40d1-a2e9-0a74dcb48286-kube-api-access-5vrnw\") pod \"monitoring-plugin-77f4f76594-rqs2c\" (UID: \"7c3b3675-ec09-40d1-a2e9-0a74dcb48286\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqs2c"
Nov 02 11:56:49.712262 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.712238    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7c3b3675-ec09-40d1-a2e9-0a74dcb48286-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rqs2c\" (UID: \"7c3b3675-ec09-40d1-a2e9-0a74dcb48286\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqs2c"
Nov 02 11:56:49.719852 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.719829    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-65867d596c-9knn9"]
Nov 02 11:56:49.721849 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.721829    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.758208 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.758144    4464 generic.go:356] "Generic (PLEG): container finished" podID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerID="3e7c6ef3a522a99e80d9e163174428a346e016a560c542664b137a774745deed" exitCode=0
Nov 02 11:56:49.758298 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.758223    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7","Type":"ContainerDied","Data":"3e7c6ef3a522a99e80d9e163174428a346e016a560c542664b137a774745deed"}
Nov 02 11:56:49.771473 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.771448    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vrnw\" (UniqueName: \"kubernetes.io/projected/7c3b3675-ec09-40d1-a2e9-0a74dcb48286-kube-api-access-5vrnw\") pod \"monitoring-plugin-77f4f76594-rqs2c\" (UID: \"7c3b3675-ec09-40d1-a2e9-0a74dcb48286\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqs2c"
Nov 02 11:56:49.775422 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.775399    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 11:56:49.775529 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.775440    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 11:56:49.775628 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.775614    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 11:56:49.775746 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.775731    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eak91j0efh19c\""
Nov 02 11:56:49.775817 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.775738    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 11:56:49.775817 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.775800    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-46jvf\""
Nov 02 11:56:49.781487 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.781465    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65867d596c-9knn9"]
Nov 02 11:56:49.810275 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.810247    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.810433 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.810410    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdc5v\" (UniqueName: \"kubernetes.io/projected/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-kube-api-access-qdc5v\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.811165 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.810697    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-client-ca-bundle\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.811165 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.810786    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-secret-metrics-server-client-certs\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.811165 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.810840    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-secret-metrics-server-tls\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.811165 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.811122    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-audit-log\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.811354 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.811189    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-metrics-server-audit-profiles\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.861165 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.861142    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqs2c"
Nov 02 11:56:49.912163 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.911919    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-secret-metrics-server-tls\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.912163 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.911946    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-audit-log\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.912163 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.911966    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-metrics-server-audit-profiles\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.912163 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.912060    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.912163 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.912088    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdc5v\" (UniqueName: \"kubernetes.io/projected/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-kube-api-access-qdc5v\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.912163 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.912135    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-client-ca-bundle\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.912813 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.912171    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-secret-metrics-server-client-certs\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.913725 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.913698    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-metrics-server-audit-profiles\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.914550 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.914505    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.914822 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.914780    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-audit-log\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.915436 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.915412    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-secret-metrics-server-tls\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.916414 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.916392    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-client-ca-bundle\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.917590 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.917572    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-secret-metrics-server-client-certs\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:49.976420 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:49.976391    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdc5v\" (UniqueName: \"kubernetes.io/projected/cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3-kube-api-access-qdc5v\") pod \"metrics-server-65867d596c-9knn9\" (UID: \"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3\") " pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:50.033991 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:50.033902    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:56:50.227644 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:50.227463    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rqs2c"]
Nov 02 11:56:50.230071 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:56:50.230025    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3b3675_ec09_40d1_a2e9_0a74dcb48286.slice/crio-629bab67ef054cedc858a9bf83d8123ae696ca65d68bbad541f9eadacc3bdfc6 WatchSource:0}: Error finding container 629bab67ef054cedc858a9bf83d8123ae696ca65d68bbad541f9eadacc3bdfc6: Status 404 returned error can't find the container with id 629bab67ef054cedc858a9bf83d8123ae696ca65d68bbad541f9eadacc3bdfc6
Nov 02 11:56:50.410202 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:50.410176    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65867d596c-9knn9"]
Nov 02 11:56:50.765867 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:50.765826    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn" event={"ID":"06d0e7da-fbf7-4c6b-b532-9ef01ab60150","Type":"ContainerStarted","Data":"37bfb9afad1fa3f1694a3ec92de3d7a2e0f0f7db4d93edca813183b23cfea72b"}
Nov 02 11:56:50.766576 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:50.765913    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn" event={"ID":"06d0e7da-fbf7-4c6b-b532-9ef01ab60150","Type":"ContainerStarted","Data":"33e40e7bde372e76bc26e81d18774f4bd897193e65fce8eb215ce9f268507f97"}
Nov 02 11:56:50.766576 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:50.765955    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn" event={"ID":"06d0e7da-fbf7-4c6b-b532-9ef01ab60150","Type":"ContainerStarted","Data":"6e6c44ae8c2bd77e68f83a3d47daa907ba274682df518bed7dfef19b37ec3b63"}
Nov 02 11:56:50.767577 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:50.767553    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65867d596c-9knn9" event={"ID":"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3","Type":"ContainerStarted","Data":"f2ed871e1a0299c09fc8c1736c7bbfaeea25721149668d85cb53dc1437b1be0f"}
Nov 02 11:56:50.768814 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:50.768789    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqs2c" event={"ID":"7c3b3675-ec09-40d1-a2e9-0a74dcb48286","Type":"ContainerStarted","Data":"629bab67ef054cedc858a9bf83d8123ae696ca65d68bbad541f9eadacc3bdfc6"}
Nov 02 11:56:52.781117 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:52.781082    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn" event={"ID":"06d0e7da-fbf7-4c6b-b532-9ef01ab60150","Type":"ContainerStarted","Data":"f4e218f2ccd610e6d60eb1ee154633aec2fb9af155e68c49deb254a46445dce5"}
Nov 02 11:56:52.781525 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:52.781179    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn" event={"ID":"06d0e7da-fbf7-4c6b-b532-9ef01ab60150","Type":"ContainerStarted","Data":"d38fffee0bbc566fe5f611cf830f86b4d6234a3b9d58e5fcd945069ce96e2126"}
Nov 02 11:56:52.781525 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:52.781231    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:56:52.781525 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:52.781262    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn" event={"ID":"06d0e7da-fbf7-4c6b-b532-9ef01ab60150","Type":"ContainerStarted","Data":"48387890e966b2ac9751079fa908b6e250ea6e9212a1daf201648b78528f43a7"}
Nov 02 11:56:52.784447 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:52.784417    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7","Type":"ContainerStarted","Data":"13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a"}
Nov 02 11:56:52.784575 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:52.784483    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7","Type":"ContainerStarted","Data":"51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f"}
Nov 02 11:56:52.784575 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:52.784525    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7","Type":"ContainerStarted","Data":"d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275"}
Nov 02 11:56:52.784575 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:52.784567    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7","Type":"ContainerStarted","Data":"d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623"}
Nov 02 11:56:52.784728 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:52.784601    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7","Type":"ContainerStarted","Data":"a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f"}
Nov 02 11:56:52.784728 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:52.784625    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7","Type":"ContainerStarted","Data":"6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3"}
Nov 02 11:56:52.785949 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:52.785918    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65867d596c-9knn9" event={"ID":"cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3","Type":"ContainerStarted","Data":"97ce4480e40284b4a5a4fb6243c31016f741f5a88eb9499ba7c5b9adc641c7f3"}
Nov 02 11:56:52.787351 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:52.787329    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqs2c" event={"ID":"7c3b3675-ec09-40d1-a2e9-0a74dcb48286","Type":"ContainerStarted","Data":"06d5852b28b7a0ad094edfb075587c027f4dca78bbf640bf7fa8cd01da96e173"}
Nov 02 11:56:52.787531 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:52.787516    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqs2c"
Nov 02 11:56:52.791945 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:52.791926    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqs2c"
Nov 02 11:56:52.937113 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:52.937066    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn" podStartSLOduration=2.26298377 podStartE2EDuration="5.937054791s" totalImagesPullingTime="3.674071021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:52.936342445 +0000 UTC m=+41.500050856" watchObservedRunningTime="2026-11-02 11:56:52.937054791 +0000 UTC m=+41.500763199"
Nov 02 11:56:53.055529 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:53.055441    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-65867d596c-9knn9" podStartSLOduration=2.075102817 podStartE2EDuration="4.055427341s" totalImagesPullingTime="1.980324524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:53.05471212 +0000 UTC m=+41.618420549" watchObservedRunningTime="2026-11-02 11:56:53.055427341 +0000 UTC m=+41.619135747"
Nov 02 11:56:53.190503 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:53.190453    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.725036497 podStartE2EDuration="6.190440328s" totalImagesPullingTime="3.45483231s" totalInitContainerRuntime="10.571521ms" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:53.189942659 +0000 UTC m=+41.753651078" watchObservedRunningTime="2026-11-02 11:56:53.190440328 +0000 UTC m=+41.754148738"
Nov 02 11:56:53.306795 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:53.306704    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rqs2c" podStartSLOduration=2.336784882 podStartE2EDuration="4.306690331s" totalImagesPullingTime="1.969905449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:53.30606296 +0000 UTC m=+41.869771372" watchObservedRunningTime="2026-11-02 11:56:53.306690331 +0000 UTC m=+41.870398739"
Nov 02 11:56:58.290822 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:56:58.290748    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 11:56:58.796706 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:56:58.796678    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7b4bb8cbdb-fh5rn"
Nov 02 11:57:10.034995 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:10.034950    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:57:10.035412 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:10.035038    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:57:28.290812 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:57:28.290776    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 11:57:30.039554 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:30.039527    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:57:30.043076 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:30.043056    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-65867d596c-9knn9"
Nov 02 11:57:38.578220 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:38.578185    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 11:57:38.578746 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:38.578525    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="alertmanager" containerID="cri-o://6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3" gracePeriod=120
Nov 02 11:57:38.578746 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:38.578572    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="kube-rbac-proxy-metric" containerID="cri-o://51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f" gracePeriod=120
Nov 02 11:57:38.578746 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:38.578610    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="kube-rbac-proxy" containerID="cri-o://d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275" gracePeriod=120
Nov 02 11:57:38.578746 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:38.578638    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="prom-label-proxy" containerID="cri-o://13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a" gracePeriod=120
Nov 02 11:57:38.578746 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:38.578571    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="kube-rbac-proxy-web" containerID="cri-o://d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623" gracePeriod=120
Nov 02 11:57:38.579018 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:38.578640    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="config-reloader" containerID="cri-o://a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f" gracePeriod=120
Nov 02 11:57:38.954423 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:38.954344    4464 generic.go:356] "Generic (PLEG): container finished" podID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerID="13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a" exitCode=0
Nov 02 11:57:38.954423 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:38.954371    4464 generic.go:356] "Generic (PLEG): container finished" podID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerID="d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275" exitCode=0
Nov 02 11:57:38.954423 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:38.954377    4464 generic.go:356] "Generic (PLEG): container finished" podID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerID="a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f" exitCode=0
Nov 02 11:57:38.954423 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:38.954382    4464 generic.go:356] "Generic (PLEG): container finished" podID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerID="6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3" exitCode=0
Nov 02 11:57:38.954648 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:38.954418    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7","Type":"ContainerDied","Data":"13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a"}
Nov 02 11:57:38.954648 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:38.954497    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7","Type":"ContainerDied","Data":"d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275"}
Nov 02 11:57:38.954648 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:38.954523    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7","Type":"ContainerDied","Data":"a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f"}
Nov 02 11:57:38.954648 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:38.954548    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7","Type":"ContainerDied","Data":"6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3"}
Nov 02 11:57:39.819197 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.819174    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:39.874929 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.874902    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-tls-assets\" (UniqueName: \"kubernetes.io/projected/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-tls-assets\") pod \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") "
Nov 02 11:57:39.875039 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.874937    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-metrics-client-ca\") pod \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") "
Nov 02 11:57:39.875039 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.874999    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-alertmanager-main-db\") pod \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") "
Nov 02 11:57:39.875039 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.875028    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-config-out\") pod \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") "
Nov 02 11:57:39.875179 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.875063    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy-web\") pod \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") "
Nov 02 11:57:39.875179 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.875093    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-web-config\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-web-config\") pod \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") "
Nov 02 11:57:39.875179 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.875133    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy\") pod \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") "
Nov 02 11:57:39.875179 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.875175    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-alertmanager-trusted-ca-bundle\") pod \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") "
Nov 02 11:57:39.875355 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.875208    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-kube-api-access-wwwl8\" (UniqueName: \"kubernetes.io/projected/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-kube-api-access-wwwl8\") pod \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") "
Nov 02 11:57:39.875355 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.875248    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-cluster-tls-config\") pod \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") "
Nov 02 11:57:39.875355 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.875278    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-main-tls\") pod \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") "
Nov 02 11:57:39.875355 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.875310    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-config-volume\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-config-volume\") pod \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") "
Nov 02 11:57:39.875355 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.875345    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\" (UID: \"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7\") "
Nov 02 11:57:39.875603 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.875375    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-metrics-client-ca" pod "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" (UID: "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 11:57:39.875603 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.875563    4464 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-metrics-client-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 11:57:39.875699 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.875596    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-alertmanager-main-db" pod "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" (UID: "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 11:57:39.876056 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.875766    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-alertmanager-trusted-ca-bundle" pod "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" (UID: "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 11:57:39.878080 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.878051    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy" pod "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" (UID: "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 11:57:39.878207 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.878182    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy-metric" pod "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" (UID: "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 11:57:39.878474 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.878447    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-main-tls" pod "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" (UID: "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 11:57:39.878544 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.878463    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-kube-api-access-wwwl8" pod "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" (UID: "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7"). InnerVolumeSpecName "kube-api-access-wwwl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 11:57:39.879260 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.879224    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-config-out" pod "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" (UID: "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 11:57:39.879260 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.879234    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-tls-assets" pod "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" (UID: "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 11:57:39.879410 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.879290    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-config-volume" pod "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" (UID: "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 11:57:39.880403 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.880384    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy-web" pod "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" (UID: "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 11:57:39.882990 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.882954    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-cluster-tls-config" pod "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" (UID: "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 11:57:39.888758 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.888713    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-web-config" pod "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" (UID: "c3d58c0d-d8a5-4074-a08d-47c32c99a5a7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 11:57:39.960523 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.960499    4464 generic.go:356] "Generic (PLEG): container finished" podID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerID="51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f" exitCode=0
Nov 02 11:57:39.960523 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.960520    4464 generic.go:356] "Generic (PLEG): container finished" podID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerID="d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623" exitCode=0
Nov 02 11:57:39.960689 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.960575    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7","Type":"ContainerDied","Data":"51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f"}
Nov 02 11:57:39.960689 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.960596    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:39.960689 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.960640    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7","Type":"ContainerDied","Data":"d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623"}
Nov 02 11:57:39.960689 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.960668    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3d58c0d-d8a5-4074-a08d-47c32c99a5a7","Type":"ContainerDied","Data":"d027bfbd2c55bf8eca5f91cc6abbdc41708f9e939f4abe14fdf737c5104e5a66"}
Nov 02 11:57:39.960897 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.960708    4464 scope.go:122] "RemoveContainer" containerID="13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a"
Nov 02 11:57:39.969052 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.969035    4464 scope.go:122] "RemoveContainer" containerID="51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f"
Nov 02 11:57:39.975492 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.975477    4464 scope.go:122] "RemoveContainer" containerID="d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275"
Nov 02 11:57:39.975873 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.975857    4464 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-tls-assets\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 11:57:39.975940 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.975877    4464 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-alertmanager-main-db\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 11:57:39.975940 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.975887    4464 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-config-out\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 11:57:39.975940 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.975896    4464 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 11:57:39.975940 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.975906    4464 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-web-config\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 11:57:39.975940 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.975915    4464 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 11:57:39.975940 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.975932    4464 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 11:57:39.976267 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.975945    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwwl8\" (UniqueName: \"kubernetes.io/projected/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-kube-api-access-wwwl8\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 11:57:39.976267 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.975956    4464 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-cluster-tls-config\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 11:57:39.976267 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.975966    4464 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-main-tls\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 11:57:39.976267 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.975991    4464 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 11:57:39.976267 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.976004    4464 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 11:57:39.982097 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.982078    4464 scope.go:122] "RemoveContainer" containerID="d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623"
Nov 02 11:57:39.988119 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.988102    4464 scope.go:122] "RemoveContainer" containerID="a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f"
Nov 02 11:57:39.994439 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:39.994422    4464 scope.go:122] "RemoveContainer" containerID="6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3"
Nov 02 11:57:40.000855 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.000838    4464 scope.go:122] "RemoveContainer" containerID="3e7c6ef3a522a99e80d9e163174428a346e016a560c542664b137a774745deed"
Nov 02 11:57:40.007005 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.006989    4464 scope.go:122] "RemoveContainer" containerID="13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a"
Nov 02 11:57:40.007240 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:57:40.007223    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a\": container with ID starting with 13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a not found: ID does not exist" containerID="13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a"
Nov 02 11:57:40.007282 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.007248    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a"} err="failed to get container status \"13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a\": rpc error: code = NotFound desc = could not find container \"13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a\": container with ID starting with 13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a not found: ID does not exist"
Nov 02 11:57:40.007282 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.007275    4464 scope.go:122] "RemoveContainer" containerID="51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f"
Nov 02 11:57:40.007497 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:57:40.007483    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f\": container with ID starting with 51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f not found: ID does not exist" containerID="51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f"
Nov 02 11:57:40.007538 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.007502    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f"} err="failed to get container status \"51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f\": rpc error: code = NotFound desc = could not find container \"51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f\": container with ID starting with 51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f not found: ID does not exist"
Nov 02 11:57:40.007538 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.007515    4464 scope.go:122] "RemoveContainer" containerID="d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275"
Nov 02 11:57:40.007724 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:57:40.007701    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275\": container with ID starting with d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275 not found: ID does not exist" containerID="d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275"
Nov 02 11:57:40.007815 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.007724    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275"} err="failed to get container status \"d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275\": rpc error: code = NotFound desc = could not find container \"d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275\": container with ID starting with d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275 not found: ID does not exist"
Nov 02 11:57:40.007815 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.007737    4464 scope.go:122] "RemoveContainer" containerID="d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623"
Nov 02 11:57:40.007954 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:57:40.007934    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623\": container with ID starting with d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623 not found: ID does not exist" containerID="d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623"
Nov 02 11:57:40.008102 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.007959    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623"} err="failed to get container status \"d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623\": rpc error: code = NotFound desc = could not find container \"d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623\": container with ID starting with d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623 not found: ID does not exist"
Nov 02 11:57:40.008102 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.007988    4464 scope.go:122] "RemoveContainer" containerID="a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f"
Nov 02 11:57:40.008260 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:57:40.008243    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f\": container with ID starting with a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f not found: ID does not exist" containerID="a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f"
Nov 02 11:57:40.008302 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.008269    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f"} err="failed to get container status \"a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f\": rpc error: code = NotFound desc = could not find container \"a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f\": container with ID starting with a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f not found: ID does not exist"
Nov 02 11:57:40.008302 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.008282    4464 scope.go:122] "RemoveContainer" containerID="6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3"
Nov 02 11:57:40.008583 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:57:40.008565    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3\": container with ID starting with 6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3 not found: ID does not exist" containerID="6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3"
Nov 02 11:57:40.008637 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.008586    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3"} err="failed to get container status \"6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3\": rpc error: code = NotFound desc = could not find container \"6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3\": container with ID starting with 6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3 not found: ID does not exist"
Nov 02 11:57:40.008637 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.008602    4464 scope.go:122] "RemoveContainer" containerID="3e7c6ef3a522a99e80d9e163174428a346e016a560c542664b137a774745deed"
Nov 02 11:57:40.008872 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:57:40.008853    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e7c6ef3a522a99e80d9e163174428a346e016a560c542664b137a774745deed\": container with ID starting with 3e7c6ef3a522a99e80d9e163174428a346e016a560c542664b137a774745deed not found: ID does not exist" containerID="3e7c6ef3a522a99e80d9e163174428a346e016a560c542664b137a774745deed"
Nov 02 11:57:40.008943 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.008876    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e7c6ef3a522a99e80d9e163174428a346e016a560c542664b137a774745deed"} err="failed to get container status \"3e7c6ef3a522a99e80d9e163174428a346e016a560c542664b137a774745deed\": rpc error: code = NotFound desc = could not find container \"3e7c6ef3a522a99e80d9e163174428a346e016a560c542664b137a774745deed\": container with ID starting with 3e7c6ef3a522a99e80d9e163174428a346e016a560c542664b137a774745deed not found: ID does not exist"
Nov 02 11:57:40.008943 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.008890    4464 scope.go:122] "RemoveContainer" containerID="13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a"
Nov 02 11:57:40.009156 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.009140    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a"} err="failed to get container status \"13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a\": rpc error: code = NotFound desc = could not find container \"13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a\": container with ID starting with 13a151f460b1d888a7ee1ba95e5f6e32120d18ee4be9a9bd01bc22c1b197c52a not found: ID does not exist"
Nov 02 11:57:40.009197 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.009156    4464 scope.go:122] "RemoveContainer" containerID="51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f"
Nov 02 11:57:40.009357 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.009342    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f"} err="failed to get container status \"51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f\": rpc error: code = NotFound desc = could not find container \"51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f\": container with ID starting with 51b18e2ffdaf9930184820d151c0814000f4ae632e6b84635af10c078c0ad53f not found: ID does not exist"
Nov 02 11:57:40.009357 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.009356    4464 scope.go:122] "RemoveContainer" containerID="d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275"
Nov 02 11:57:40.009560 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.009539    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275"} err="failed to get container status \"d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275\": rpc error: code = NotFound desc = could not find container \"d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275\": container with ID starting with d8996cfe2331f7b5a92d34ab1985363666bb39dd52dcc5d3535a483ffe35a275 not found: ID does not exist"
Nov 02 11:57:40.009601 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.009561    4464 scope.go:122] "RemoveContainer" containerID="d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623"
Nov 02 11:57:40.009800 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.009784    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623"} err="failed to get container status \"d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623\": rpc error: code = NotFound desc = could not find container \"d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623\": container with ID starting with d072b0c39d36c372f36bee6c89509dfd17182f4eb30b08623bcaddb9ed595623 not found: ID does not exist"
Nov 02 11:57:40.009840 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.009800    4464 scope.go:122] "RemoveContainer" containerID="a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f"
Nov 02 11:57:40.010013 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.009995    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f"} err="failed to get container status \"a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f\": rpc error: code = NotFound desc = could not find container \"a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f\": container with ID starting with a5f3701b53c6eadd79a7f0c37fa6396dc5fbdfef01b624b84c3af2ea7017e87f not found: ID does not exist"
Nov 02 11:57:40.010059 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.010013    4464 scope.go:122] "RemoveContainer" containerID="6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3"
Nov 02 11:57:40.010234 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.010217    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3"} err="failed to get container status \"6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3\": rpc error: code = NotFound desc = could not find container \"6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3\": container with ID starting with 6e506ba0339b2527fffd4af37a94bfcf63eb6f22224486029b20e690016071f3 not found: ID does not exist"
Nov 02 11:57:40.010274 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.010234    4464 scope.go:122] "RemoveContainer" containerID="3e7c6ef3a522a99e80d9e163174428a346e016a560c542664b137a774745deed"
Nov 02 11:57:40.010426 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.010411    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e7c6ef3a522a99e80d9e163174428a346e016a560c542664b137a774745deed"} err="failed to get container status \"3e7c6ef3a522a99e80d9e163174428a346e016a560c542664b137a774745deed\": rpc error: code = NotFound desc = could not find container \"3e7c6ef3a522a99e80d9e163174428a346e016a560c542664b137a774745deed\": container with ID starting with 3e7c6ef3a522a99e80d9e163174428a346e016a560c542664b137a774745deed not found: ID does not exist"
Nov 02 11:57:40.217436 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.217362    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 11:57:40.221693 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.221672    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 11:57:40.267704 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.267678    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 11:57:40.267950 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.267938    4464 cpu_manager.go:414] "containerMap: removing container" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="config-reloader"
Nov 02 11:57:40.268016 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.267951    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="config-reloader"
Nov 02 11:57:40.268016 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.267963    4464 cpu_manager.go:414] "containerMap: removing container" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="alertmanager"
Nov 02 11:57:40.268016 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.267968    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="alertmanager"
Nov 02 11:57:40.268016 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.267993    4464 cpu_manager.go:414] "containerMap: removing container" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="prom-label-proxy"
Nov 02 11:57:40.268016 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.268001    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="prom-label-proxy"
Nov 02 11:57:40.268016 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.268011    4464 cpu_manager.go:414] "containerMap: removing container" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="kube-rbac-proxy-metric"
Nov 02 11:57:40.268016 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.268016    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="kube-rbac-proxy-metric"
Nov 02 11:57:40.268224 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.268031    4464 cpu_manager.go:414] "containerMap: removing container" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="kube-rbac-proxy-web"
Nov 02 11:57:40.268224 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.268036    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="kube-rbac-proxy-web"
Nov 02 11:57:40.268224 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.268042    4464 cpu_manager.go:414] "containerMap: removing container" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="kube-rbac-proxy"
Nov 02 11:57:40.268224 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.268047    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="kube-rbac-proxy"
Nov 02 11:57:40.268224 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.268055    4464 cpu_manager.go:414] "containerMap: removing container" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="init-config-reloader"
Nov 02 11:57:40.268224 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.268059    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="init-config-reloader"
Nov 02 11:57:40.268224 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.268112    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="config-reloader"
Nov 02 11:57:40.268224 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.268120    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="alertmanager"
Nov 02 11:57:40.268224 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.268127    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="prom-label-proxy"
Nov 02 11:57:40.268224 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.268133    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="kube-rbac-proxy-metric"
Nov 02 11:57:40.268224 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.268140    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="kube-rbac-proxy-web"
Nov 02 11:57:40.268224 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.268148    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" containerName="kube-rbac-proxy"
Nov 02 11:57:40.271554 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.271538    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.324941 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.324911    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 11:57:40.324941 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.324925    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 11:57:40.324941 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.324926    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 11:57:40.325179 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.325032    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 11:57:40.325179 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.325032    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 11:57:40.325179 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.325097    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 11:57:40.325289 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.325198    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 11:57:40.325372 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.325346    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 11:57:40.325506 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.325487    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bbkkg\""
Nov 02 11:57:40.375419 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.375376    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 11:57:40.376346 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.376329    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 11:57:40.378334 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.378318    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84a78b98-6493-41d0-9d0a-692504bd16c7-config-out\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.378397 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.378342    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84a78b98-6493-41d0-9d0a-692504bd16c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.378397 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.378364    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84a78b98-6493-41d0-9d0a-692504bd16c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.378397 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.378379    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4mpz\" (UniqueName: \"kubernetes.io/projected/84a78b98-6493-41d0-9d0a-692504bd16c7-kube-api-access-p4mpz\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.378516 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.378452    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84a78b98-6493-41d0-9d0a-692504bd16c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.378516 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.378505    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.378591 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.378546    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.378591 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.378569    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.378674 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.378594    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.378674 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.378617    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.378776 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.378667    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.378776 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.378713    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-web-config\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.378838 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.378783    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84a78b98-6493-41d0-9d0a-692504bd16c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.479191 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.479167    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.479307 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.479199    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.479307 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.479222    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.479449 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.479351    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.479449 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.479385    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.479449 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.479407    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-web-config\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.479596 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.479521    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84a78b98-6493-41d0-9d0a-692504bd16c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.479596 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.479584    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84a78b98-6493-41d0-9d0a-692504bd16c7-config-out\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.479714 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.479616    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84a78b98-6493-41d0-9d0a-692504bd16c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.479714 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.479663    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84a78b98-6493-41d0-9d0a-692504bd16c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.479714 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.479686    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4mpz\" (UniqueName: \"kubernetes.io/projected/84a78b98-6493-41d0-9d0a-692504bd16c7-kube-api-access-p4mpz\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.479854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.479739    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84a78b98-6493-41d0-9d0a-692504bd16c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.479854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.479765    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.481064 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.480726    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84a78b98-6493-41d0-9d0a-692504bd16c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.481064 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.480970    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84a78b98-6493-41d0-9d0a-692504bd16c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.481380 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.481356    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84a78b98-6493-41d0-9d0a-692504bd16c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.482270 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.482219    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.482354 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.482305    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.482421 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.482384    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-web-config\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.482484 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.482420    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.482484 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.482460    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.482937 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.482912    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.483343 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.483326    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84a78b98-6493-41d0-9d0a-692504bd16c7-config-out\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.483653 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.483629    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84a78b98-6493-41d0-9d0a-692504bd16c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.483732 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.483687    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84a78b98-6493-41d0-9d0a-692504bd16c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.541635 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.541612    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4mpz\" (UniqueName: \"kubernetes.io/projected/84a78b98-6493-41d0-9d0a-692504bd16c7-kube-api-access-p4mpz\") pod \"alertmanager-main-0\" (UID: \"84a78b98-6493-41d0-9d0a-692504bd16c7\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.587274 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.587250    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 11:57:40.842126 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.842059    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-vfccn-qnzjq"]
Nov 02 11:57:40.842470 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.842250    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-vfccn-qnzjq" podUID="96b40a95-1bfc-4388-b366-f5464623ddb5" containerName="insights-gathering" containerID="cri-o://491d92ca92d6f67a078b6d7d3338af189b202ee9658f50af9acd79c2813ded58" gracePeriod=30
Nov 02 11:57:40.966147 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.966123    4464 generic.go:356] "Generic (PLEG): container finished" podID="96b40a95-1bfc-4388-b366-f5464623ddb5" containerID="491d92ca92d6f67a078b6d7d3338af189b202ee9658f50af9acd79c2813ded58" exitCode=0
Nov 02 11:57:40.966259 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.966179    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vfccn-qnzjq" event={"ID":"96b40a95-1bfc-4388-b366-f5464623ddb5","Type":"ContainerDied","Data":"491d92ca92d6f67a078b6d7d3338af189b202ee9658f50af9acd79c2813ded58"}
Nov 02 11:57:40.966259 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.966211    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vfccn-qnzjq" event={"ID":"96b40a95-1bfc-4388-b366-f5464623ddb5","Type":"ContainerDied","Data":"5fa794ba481e5c1ece7bb9d2926631d05c08d79d5d4f6f8dd44c9e2217ffa826"}
Nov 02 11:57:40.966259 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.966227    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fa794ba481e5c1ece7bb9d2926631d05c08d79d5d4f6f8dd44c9e2217ffa826"
Nov 02 11:57:40.974365 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:40.974343    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:57:41.011194 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.011173    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 11:57:41.013790 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:57:41.013768    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a78b98_6493_41d0_9d0a_692504bd16c7.slice/crio-d5f470402fbe0e3d9aa6b4174fc8a9d30ffbf440bee250c4d59600ac5a20499a WatchSource:0}: Error finding container d5f470402fbe0e3d9aa6b4174fc8a9d30ffbf440bee250c4d59600ac5a20499a: Status 404 returned error can't find the container with id d5f470402fbe0e3d9aa6b4174fc8a9d30ffbf440bee250c4d59600ac5a20499a
Nov 02 11:57:41.084148 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.084126    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b40a95-1bfc-4388-b366-f5464623ddb5-kube-api-access-vjrdq\" (UniqueName: \"kubernetes.io/projected/96b40a95-1bfc-4388-b366-f5464623ddb5-kube-api-access-vjrdq\") pod \"96b40a95-1bfc-4388-b366-f5464623ddb5\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") "
Nov 02 11:57:41.084230 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.084168    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-trusted-ca-bundle\") pod \"96b40a95-1bfc-4388-b366-f5464623ddb5\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") "
Nov 02 11:57:41.084230 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.084198    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-service-ca-bundle\") pod \"96b40a95-1bfc-4388-b366-f5464623ddb5\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") "
Nov 02 11:57:41.084306 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.084262    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b40a95-1bfc-4388-b366-f5464623ddb5-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/96b40a95-1bfc-4388-b366-f5464623ddb5-archives-path\") pod \"96b40a95-1bfc-4388-b366-f5464623ddb5\" (UID: \"96b40a95-1bfc-4388-b366-f5464623ddb5\") "
Nov 02 11:57:41.084578 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.084555    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-service-ca-bundle" pod "96b40a95-1bfc-4388-b366-f5464623ddb5" (UID: "96b40a95-1bfc-4388-b366-f5464623ddb5"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 11:57:41.084649 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.084575    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b40a95-1bfc-4388-b366-f5464623ddb5-archives-path" pod "96b40a95-1bfc-4388-b366-f5464623ddb5" (UID: "96b40a95-1bfc-4388-b366-f5464623ddb5"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 11:57:41.084649 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.084586    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-trusted-ca-bundle" pod "96b40a95-1bfc-4388-b366-f5464623ddb5" (UID: "96b40a95-1bfc-4388-b366-f5464623ddb5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 11:57:41.086142 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.086124    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b40a95-1bfc-4388-b366-f5464623ddb5-kube-api-access-vjrdq" pod "96b40a95-1bfc-4388-b366-f5464623ddb5" (UID: "96b40a95-1bfc-4388-b366-f5464623ddb5"). InnerVolumeSpecName "kube-api-access-vjrdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 11:57:41.185481 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.185460    4464 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/96b40a95-1bfc-4388-b366-f5464623ddb5-archives-path\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 11:57:41.185481 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.185479    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjrdq\" (UniqueName: \"kubernetes.io/projected/96b40a95-1bfc-4388-b366-f5464623ddb5-kube-api-access-vjrdq\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 11:57:41.185609 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.185490    4464 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-trusted-ca-bundle\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 11:57:41.185609 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.185499    4464 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96b40a95-1bfc-4388-b366-f5464623ddb5-service-ca-bundle\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 11:57:41.579064 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.579034    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3d58c0d-d8a5-4074-a08d-47c32c99a5a7" path="/var/lib/kubelet/pods/c3d58c0d-d8a5-4074-a08d-47c32c99a5a7/volumes"
Nov 02 11:57:41.970519 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.970492    4464 generic.go:356] "Generic (PLEG): container finished" podID="84a78b98-6493-41d0-9d0a-692504bd16c7" containerID="65f87cd620fb77421521dace07063a087b291a05b8c5a25922521dc7e68dbac5" exitCode=0
Nov 02 11:57:41.970885 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.970574    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84a78b98-6493-41d0-9d0a-692504bd16c7","Type":"ContainerDied","Data":"65f87cd620fb77421521dace07063a087b291a05b8c5a25922521dc7e68dbac5"}
Nov 02 11:57:41.970885 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.970603    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vfccn-qnzjq"
Nov 02 11:57:41.970885 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:41.970636    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84a78b98-6493-41d0-9d0a-692504bd16c7","Type":"ContainerStarted","Data":"d5f470402fbe0e3d9aa6b4174fc8a9d30ffbf440bee250c4d59600ac5a20499a"}
Nov 02 11:57:42.212082 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.212057    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-vfccn-qnzjq"]
Nov 02 11:57:42.216105 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.216084    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-vfccn-qnzjq"]
Nov 02 11:57:42.290515 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.290493    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"]
Nov 02 11:57:42.290808 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.290796    4464 cpu_manager.go:414] "containerMap: removing container" podUID="96b40a95-1bfc-4388-b366-f5464623ddb5" containerName="insights-gathering"
Nov 02 11:57:42.291043 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.290810    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b40a95-1bfc-4388-b366-f5464623ddb5" containerName="insights-gathering"
Nov 02 11:57:42.291043 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.290867    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b40a95-1bfc-4388-b366-f5464623ddb5" containerName="insights-gathering"
Nov 02 11:57:42.292624 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.292609    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.346166 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.346139    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 02 11:57:42.346277 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.346185    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 02 11:57:42.346277 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.346237    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 02 11:57:42.346390 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.346359    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 02 11:57:42.346590 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.346572    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-p6pdf\""
Nov 02 11:57:42.346794 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.346780    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 02 11:57:42.395633 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.395604    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1d48ddc2-9b07-448a-b214-9aadd38d54b4-telemeter-client-tls\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.395732 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.395644    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d48ddc2-9b07-448a-b214-9aadd38d54b4-metrics-client-ca\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.395802 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.395732    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1d48ddc2-9b07-448a-b214-9aadd38d54b4-federate-client-tls\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.395839 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.395819    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g49f9\" (UniqueName: \"kubernetes.io/projected/1d48ddc2-9b07-448a-b214-9aadd38d54b4-kube-api-access-g49f9\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.395899 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.395882    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d48ddc2-9b07-448a-b214-9aadd38d54b4-serving-certs-ca-bundle\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.396189 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.396013    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1d48ddc2-9b07-448a-b214-9aadd38d54b4-secret-telemeter-client\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.396189 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.396050    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d48ddc2-9b07-448a-b214-9aadd38d54b4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.396189 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.396128    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d48ddc2-9b07-448a-b214-9aadd38d54b4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.397704 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.397683    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 02 11:57:42.398728 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.398700    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"]
Nov 02 11:57:42.496854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.496792    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d48ddc2-9b07-448a-b214-9aadd38d54b4-serving-certs-ca-bundle\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.496854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.496820    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1d48ddc2-9b07-448a-b214-9aadd38d54b4-secret-telemeter-client\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.496854 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.496838    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d48ddc2-9b07-448a-b214-9aadd38d54b4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.497077 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.496954    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d48ddc2-9b07-448a-b214-9aadd38d54b4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.497077 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.497033    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1d48ddc2-9b07-448a-b214-9aadd38d54b4-telemeter-client-tls\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.497077 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.497065    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d48ddc2-9b07-448a-b214-9aadd38d54b4-metrics-client-ca\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.497194 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.497108    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1d48ddc2-9b07-448a-b214-9aadd38d54b4-federate-client-tls\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.497194 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.497149    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g49f9\" (UniqueName: \"kubernetes.io/projected/1d48ddc2-9b07-448a-b214-9aadd38d54b4-kube-api-access-g49f9\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.497597 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.497558    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d48ddc2-9b07-448a-b214-9aadd38d54b4-serving-certs-ca-bundle\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.497750 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.497728    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d48ddc2-9b07-448a-b214-9aadd38d54b4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.497902 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.497884    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d48ddc2-9b07-448a-b214-9aadd38d54b4-metrics-client-ca\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.499638 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.499615    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1d48ddc2-9b07-448a-b214-9aadd38d54b4-federate-client-tls\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.499725 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.499676    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1d48ddc2-9b07-448a-b214-9aadd38d54b4-telemeter-client-tls\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.499814 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.499795    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d48ddc2-9b07-448a-b214-9aadd38d54b4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.499864 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.499846    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1d48ddc2-9b07-448a-b214-9aadd38d54b4-secret-telemeter-client\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.558691 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.558665    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g49f9\" (UniqueName: \"kubernetes.io/projected/1d48ddc2-9b07-448a-b214-9aadd38d54b4-kube-api-access-g49f9\") pod \"telemeter-client-cdf8c8bbd-b64wq\" (UID: \"1d48ddc2-9b07-448a-b214-9aadd38d54b4\") " pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.604011 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.603992    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"
Nov 02 11:57:42.979735 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.979699    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84a78b98-6493-41d0-9d0a-692504bd16c7","Type":"ContainerStarted","Data":"8f8c851328ba4775b457eb1c4a8a2d3d5338bf24f8b2db4d53e2b599108145df"}
Nov 02 11:57:42.980111 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.979805    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84a78b98-6493-41d0-9d0a-692504bd16c7","Type":"ContainerStarted","Data":"383c1372485e1aabf746960ef1589fe447e598016cb447232ca41108fdc73c69"}
Nov 02 11:57:42.980111 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.979847    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84a78b98-6493-41d0-9d0a-692504bd16c7","Type":"ContainerStarted","Data":"374bd98f7d5954763f12bde9a755a1a11ffbe13dac3a9cf0500cba756d36beef"}
Nov 02 11:57:42.980111 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.979890    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84a78b98-6493-41d0-9d0a-692504bd16c7","Type":"ContainerStarted","Data":"9b6d3b6d89fd7fcaa5ba21c47f6e9e02e98b292db2572c3d6e3ef095787df465"}
Nov 02 11:57:42.980111 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.979925    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84a78b98-6493-41d0-9d0a-692504bd16c7","Type":"ContainerStarted","Data":"2db41843401c5dbda05bebb36003bc02a70bff3a8a2d477837deb5d4d1139320"}
Nov 02 11:57:42.980111 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:42.979962    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84a78b98-6493-41d0-9d0a-692504bd16c7","Type":"ContainerStarted","Data":"fe7175153f27349a0156c1f16613dc8ca70a90b5689c6121512e390e3675d5dd"}
Nov 02 11:57:43.027676 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:43.027488    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq"]
Nov 02 11:57:43.029944 ip-10-0-133-239 kubenswrapper[4464]: W1102 11:57:43.029918    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d48ddc2_9b07_448a_b214_9aadd38d54b4.slice/crio-5f2e1adc2052590950026eba0ca680973acde5f9d7f8332102d9e405c356eb66 WatchSource:0}: Error finding container 5f2e1adc2052590950026eba0ca680973acde5f9d7f8332102d9e405c356eb66: Status 404 returned error can't find the container with id 5f2e1adc2052590950026eba0ca680973acde5f9d7f8332102d9e405c356eb66
Nov 02 11:57:43.112483 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:43.112440    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.094226538 podStartE2EDuration="3.11242833s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.201792ms" isStatefulPod=false podCreationTimestamp="2026-11-02 11:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:57:43.111428683 +0000 UTC m=+91.675137091" watchObservedRunningTime="2026-11-02 11:57:43.11242833 +0000 UTC m=+91.676136739"
Nov 02 11:57:43.579761 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:43.579729    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b40a95-1bfc-4388-b366-f5464623ddb5" path="/var/lib/kubelet/pods/96b40a95-1bfc-4388-b366-f5464623ddb5/volumes"
Nov 02 11:57:43.985136 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:43.985098    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq" event={"ID":"1d48ddc2-9b07-448a-b214-9aadd38d54b4","Type":"ContainerStarted","Data":"5f2e1adc2052590950026eba0ca680973acde5f9d7f8332102d9e405c356eb66"}
Nov 02 11:57:44.990034 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:44.990005    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq" event={"ID":"1d48ddc2-9b07-448a-b214-9aadd38d54b4","Type":"ContainerStarted","Data":"2cb735ce2d6a7710f7f21edaafdf1ebc0c098b071deef6936a935f0abdb96f65"}
Nov 02 11:57:44.990336 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:44.990060    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq" event={"ID":"1d48ddc2-9b07-448a-b214-9aadd38d54b4","Type":"ContainerStarted","Data":"db2bae2e457afa1a664afe31aed6ce42dc4dcfdb6216355d1f5a04518b90af7c"}
Nov 02 11:57:45.996121 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:45.996086    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq" event={"ID":"1d48ddc2-9b07-448a-b214-9aadd38d54b4","Type":"ContainerStarted","Data":"34f76e7521136418a8e82932bb8548a1686cfbbba4e813b51aebdf00ee8a2d19"}
Nov 02 11:57:46.223935 ip-10-0-133-239 kubenswrapper[4464]: I1102 11:57:46.223876    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-cdf8c8bbd-b64wq" podStartSLOduration=2.406822534 podStartE2EDuration="4.223857662s" totalImagesPullingTime="1.817035128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:57:46.223119799 +0000 UTC m=+94.786828249" watchObservedRunningTime="2026-11-02 11:57:46.223857662 +0000 UTC m=+94.787566072"
Nov 02 11:57:58.290719 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:57:58.290672    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 11:58:28.291022 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:58:28.290920    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 11:58:58.290526 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:58:58.290483    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 11:59:28.289746 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:59:28.289702    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 11:59:58.290746 ip-10-0-133-239 kubenswrapper[4464]: E1102 11:59:58.290663    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:00:28.291222 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:00:28.291186    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:00:58.290682 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:00:58.290641    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:01:11.582307 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:11.582277    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:01:11.583310 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:11.583287    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:01:17.904752 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:17.904718    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd"]
Nov 02 12:01:17.908145 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:17.908125    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd"
Nov 02 12:01:17.962107 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:17.962074    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-xdj85\""
Nov 02 12:01:17.990667 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:17.990642    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 12:01:17.990861 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:17.990843    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 12:01:18.013811 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.013787    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd"]
Nov 02 12:01:18.014498 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.014481    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt"]
Nov 02 12:01:18.018016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.018002    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt"
Nov 02 12:01:18.024236 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.024211    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd\" (UID: \"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd"
Nov 02 12:01:18.024305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.024276    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqfjz\" (UniqueName: \"kubernetes.io/projected/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-kube-api-access-sqfjz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd\" (UID: \"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd"
Nov 02 12:01:18.024362 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.024331    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd\" (UID: \"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd"
Nov 02 12:01:18.071405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.071381    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-x42jj\""
Nov 02 12:01:18.091122 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.091096    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt"]
Nov 02 12:01:18.125450 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.125423    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg7dp\" (UniqueName: \"kubernetes.io/projected/6626f409-46a9-4003-85dc-3d6c53059674-kube-api-access-jg7dp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt\" (UID: \"6626f409-46a9-4003-85dc-3d6c53059674\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt"
Nov 02 12:01:18.125550 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.125458    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd\" (UID: \"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd"
Nov 02 12:01:18.125550 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.125532    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqfjz\" (UniqueName: \"kubernetes.io/projected/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-kube-api-access-sqfjz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd\" (UID: \"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd"
Nov 02 12:01:18.125621 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.125564    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6626f409-46a9-4003-85dc-3d6c53059674-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt\" (UID: \"6626f409-46a9-4003-85dc-3d6c53059674\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt"
Nov 02 12:01:18.125621 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.125613    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6626f409-46a9-4003-85dc-3d6c53059674-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt\" (UID: \"6626f409-46a9-4003-85dc-3d6c53059674\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt"
Nov 02 12:01:18.125707 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.125682    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd\" (UID: \"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd"
Nov 02 12:01:18.125904 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.125889    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd\" (UID: \"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd"
Nov 02 12:01:18.126043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.126027    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd\" (UID: \"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd"
Nov 02 12:01:18.187915 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.187832    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqfjz\" (UniqueName: \"kubernetes.io/projected/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-kube-api-access-sqfjz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd\" (UID: \"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd"
Nov 02 12:01:18.226342 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.226313    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg7dp\" (UniqueName: \"kubernetes.io/projected/6626f409-46a9-4003-85dc-3d6c53059674-kube-api-access-jg7dp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt\" (UID: \"6626f409-46a9-4003-85dc-3d6c53059674\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt"
Nov 02 12:01:18.226471 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.226366    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6626f409-46a9-4003-85dc-3d6c53059674-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt\" (UID: \"6626f409-46a9-4003-85dc-3d6c53059674\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt"
Nov 02 12:01:18.226471 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.226391    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6626f409-46a9-4003-85dc-3d6c53059674-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt\" (UID: \"6626f409-46a9-4003-85dc-3d6c53059674\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt"
Nov 02 12:01:18.226695 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.226680    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6626f409-46a9-4003-85dc-3d6c53059674-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt\" (UID: \"6626f409-46a9-4003-85dc-3d6c53059674\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt"
Nov 02 12:01:18.226740 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.226710    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6626f409-46a9-4003-85dc-3d6c53059674-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt\" (UID: \"6626f409-46a9-4003-85dc-3d6c53059674\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt"
Nov 02 12:01:18.227217 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.227191    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd"
Nov 02 12:01:18.243922 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.243897    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"]
Nov 02 12:01:18.247545 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.247530    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.290633 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.290605    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg7dp\" (UniqueName: \"kubernetes.io/projected/6626f409-46a9-4003-85dc-3d6c53059674-kube-api-access-jg7dp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt\" (UID: \"6626f409-46a9-4003-85dc-3d6c53059674\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt"
Nov 02 12:01:18.306281 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.306231    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 12:01:18.306931 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.306912    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 12:01:18.308104 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.308085    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 12:01:18.308176 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.308147    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 12:01:18.308176 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.308164    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-j8d6n\""
Nov 02 12:01:18.308281 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.308269    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 12:01:18.311336 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.311311    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"]
Nov 02 12:01:18.333365 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.333342    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt"
Nov 02 12:01:18.406642 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.406619    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 12:01:18.428106 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.428083    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/409d846e-dfe5-45c5-a672-d4bc9513d5b2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.428210 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.428112    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.428210 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.428139    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf5th\" (UniqueName: \"kubernetes.io/projected/409d846e-dfe5-45c5-a672-d4bc9513d5b2-kube-api-access-zf5th\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.428284 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.428242    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.428284 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.428263    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.428284 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.428280    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.428418 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.428305    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.529295 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.529261    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5th\" (UniqueName: \"kubernetes.io/projected/409d846e-dfe5-45c5-a672-d4bc9513d5b2-kube-api-access-zf5th\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.529439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.529316    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.529439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.529343    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.529439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.529361    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.529439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.529384    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.529439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.529427    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/409d846e-dfe5-45c5-a672-d4bc9513d5b2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.529680 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.529444    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.530127 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.530102    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.531577 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.531560    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/409d846e-dfe5-45c5-a672-d4bc9513d5b2-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.531739 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.531723    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.532149 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.532128    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.532226 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.532127    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.592678 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.592652    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf5th\" (UniqueName: \"kubernetes.io/projected/409d846e-dfe5-45c5-a672-d4bc9513d5b2-kube-api-access-zf5th\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.592774 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.592757    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-bwld5\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:18.601787 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.601768    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd"]
Nov 02 12:01:18.603843 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:01:18.603823    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a2d0dd_eede_4f2e_8ff5_2a03c70d74a9.slice/crio-e1aa305d9a564a2f53d23a190cb3ce95c2454288e87afb56fae5e16e14431000 WatchSource:0}: Error finding container e1aa305d9a564a2f53d23a190cb3ce95c2454288e87afb56fae5e16e14431000: Status 404 returned error can't find the container with id e1aa305d9a564a2f53d23a190cb3ce95c2454288e87afb56fae5e16e14431000
Nov 02 12:01:18.605717 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.605701    4464 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:01:18.712301 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.712282    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt"]
Nov 02 12:01:18.714451 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:01:18.714429    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6626f409_46a9_4003_85dc_3d6c53059674.slice/crio-9268c73fbb204ae781703a2345119623c17c6fbeb4280743dc822bd24344e9d0 WatchSource:0}: Error finding container 9268c73fbb204ae781703a2345119623c17c6fbeb4280743dc822bd24344e9d0: Status 404 returned error can't find the container with id 9268c73fbb204ae781703a2345119623c17c6fbeb4280743dc822bd24344e9d0
Nov 02 12:01:18.730315 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.730285    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd" event={"ID":"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9","Type":"ContainerStarted","Data":"e1aa305d9a564a2f53d23a190cb3ce95c2454288e87afb56fae5e16e14431000"}
Nov 02 12:01:18.731418 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.731399    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt" event={"ID":"6626f409-46a9-4003-85dc-3d6c53059674","Type":"ContainerStarted","Data":"9268c73fbb204ae781703a2345119623c17c6fbeb4280743dc822bd24344e9d0"}
Nov 02 12:01:18.858627 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:18.858542    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:19.244111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:19.244089    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"]
Nov 02 12:01:19.246566 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:01:19.246524    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409d846e_dfe5_45c5_a672_d4bc9513d5b2.slice/crio-08c3c08b8970d10af03b759a66d07ec81f90e8d5bf60bbd0936c23d0af7055cc WatchSource:0}: Error finding container 08c3c08b8970d10af03b759a66d07ec81f90e8d5bf60bbd0936c23d0af7055cc: Status 404 returned error can't find the container with id 08c3c08b8970d10af03b759a66d07ec81f90e8d5bf60bbd0936c23d0af7055cc
Nov 02 12:01:19.743263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:19.743222    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5" event={"ID":"409d846e-dfe5-45c5-a672-d4bc9513d5b2","Type":"ContainerStarted","Data":"08c3c08b8970d10af03b759a66d07ec81f90e8d5bf60bbd0936c23d0af7055cc"}
Nov 02 12:01:23.862779 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:23.862247    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:01:23.862779 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:23.862313    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:01:24.774169 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:24.773962    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5" event={"ID":"409d846e-dfe5-45c5-a672-d4bc9513d5b2","Type":"ContainerStarted","Data":"b27e6ac3ecbc6ac814b34ad4cf55f9ed1783694a87d5b20d7f4a85a5afb41229"}
Nov 02 12:01:24.774881 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:24.774857    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:24.776924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:24.776648    4464 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-bwld5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:01:24.776924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:24.776698    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5" podUID="409d846e-dfe5-45c5-a672-d4bc9513d5b2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:01:24.998821 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:24.998763    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5" podStartSLOduration=2.385975083 podStartE2EDuration="6.998745667s" totalImagesPullingTime="4.612770584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:01:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:01:24.947758874 +0000 UTC m=+313.511467295" watchObservedRunningTime="2026-11-02 12:01:24.998745667 +0000 UTC m=+313.562454077"
Nov 02 12:01:25.778930 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:25.778899    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:01:26.782784 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:26.782742    4464 generic.go:356] "Generic (PLEG): container finished" podID="97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9" containerID="6a82007341b942ddcf77b3c21dd5638bd2bfef14c8cd90610382e7fe90e6e270" exitCode=0
Nov 02 12:01:26.783248 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:26.782835    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd" event={"ID":"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9","Type":"ContainerDied","Data":"6a82007341b942ddcf77b3c21dd5638bd2bfef14c8cd90610382e7fe90e6e270"}
Nov 02 12:01:26.784454 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:26.784430    4464 generic.go:356] "Generic (PLEG): container finished" podID="6626f409-46a9-4003-85dc-3d6c53059674" containerID="4d78a1a9c2c7624ba5ef30f9cbceac5650a8dcf46c5331dfc2abc8255dd2d463" exitCode=0
Nov 02 12:01:26.784572 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:26.784545    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt" event={"ID":"6626f409-46a9-4003-85dc-3d6c53059674","Type":"ContainerDied","Data":"4d78a1a9c2c7624ba5ef30f9cbceac5650a8dcf46c5331dfc2abc8255dd2d463"}
Nov 02 12:01:28.289655 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:01:28.289617    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:01:28.796441 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:28.796410    4464 generic.go:356] "Generic (PLEG): container finished" podID="6626f409-46a9-4003-85dc-3d6c53059674" containerID="350e3e77e63de770e2f0e93505c3fe012e6e2024c22914a015a0bce728ed5a21" exitCode=0
Nov 02 12:01:28.796628 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:28.796449    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt" event={"ID":"6626f409-46a9-4003-85dc-3d6c53059674","Type":"ContainerDied","Data":"350e3e77e63de770e2f0e93505c3fe012e6e2024c22914a015a0bce728ed5a21"}
Nov 02 12:01:30.806345 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:30.806257    4464 generic.go:356] "Generic (PLEG): container finished" podID="97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9" containerID="bf65f17894fa22905b2e4d9bfc576d188f4bbf193767e331a38dc2cc6810054f" exitCode=0
Nov 02 12:01:30.806766 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:30.806356    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd" event={"ID":"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9","Type":"ContainerDied","Data":"bf65f17894fa22905b2e4d9bfc576d188f4bbf193767e331a38dc2cc6810054f"}
Nov 02 12:01:35.829023 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:35.828964    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt" event={"ID":"6626f409-46a9-4003-85dc-3d6c53059674","Type":"ContainerStarted","Data":"fa80492e30f7bfc93e432f5592d81667d88db82057cbd7afd7f396458a041114"}
Nov 02 12:01:35.954115 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:35.953999    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt" podStartSLOduration=4.074765484 podStartE2EDuration="18.953960453s" totalImagesPullingTime="14.879194969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:01:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:01:35.952824238 +0000 UTC m=+324.516532647" watchObservedRunningTime="2026-11-02 12:01:35.953960453 +0000 UTC m=+324.517668863"
Nov 02 12:01:36.835349 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:36.835312    4464 generic.go:356] "Generic (PLEG): container finished" podID="6626f409-46a9-4003-85dc-3d6c53059674" containerID="fa80492e30f7bfc93e432f5592d81667d88db82057cbd7afd7f396458a041114" exitCode=0
Nov 02 12:01:36.835770 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:36.835390    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt" event={"ID":"6626f409-46a9-4003-85dc-3d6c53059674","Type":"ContainerDied","Data":"fa80492e30f7bfc93e432f5592d81667d88db82057cbd7afd7f396458a041114"}
Nov 02 12:01:37.841530 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:37.841446    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd" event={"ID":"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9","Type":"ContainerStarted","Data":"2d9e0f79976e0482a5d844917479ab2c9447134bf2f584fc67f7bc36fb7c6ab9"}
Nov 02 12:01:37.961392 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:37.961345    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd" podStartSLOduration=3.281376209 podStartE2EDuration="20.961329864s" totalImagesPullingTime="17.679953655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:01:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:01:37.960549839 +0000 UTC m=+326.524258249" watchObservedRunningTime="2026-11-02 12:01:37.961329864 +0000 UTC m=+326.525038269"
Nov 02 12:01:37.963274 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:37.963255    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt"
Nov 02 12:01:38.008687 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:38.008656    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6626f409-46a9-4003-85dc-3d6c53059674-kube-api-access-jg7dp\" (UniqueName: \"kubernetes.io/projected/6626f409-46a9-4003-85dc-3d6c53059674-kube-api-access-jg7dp\") pod \"6626f409-46a9-4003-85dc-3d6c53059674\" (UID: \"6626f409-46a9-4003-85dc-3d6c53059674\") "
Nov 02 12:01:38.008816 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:38.008750    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6626f409-46a9-4003-85dc-3d6c53059674-util\" (UniqueName: \"kubernetes.io/empty-dir/6626f409-46a9-4003-85dc-3d6c53059674-util\") pod \"6626f409-46a9-4003-85dc-3d6c53059674\" (UID: \"6626f409-46a9-4003-85dc-3d6c53059674\") "
Nov 02 12:01:38.008816 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:38.008792    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6626f409-46a9-4003-85dc-3d6c53059674-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6626f409-46a9-4003-85dc-3d6c53059674-bundle\") pod \"6626f409-46a9-4003-85dc-3d6c53059674\" (UID: \"6626f409-46a9-4003-85dc-3d6c53059674\") "
Nov 02 12:01:38.009352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:38.009328    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6626f409-46a9-4003-85dc-3d6c53059674-bundle" pod "6626f409-46a9-4003-85dc-3d6c53059674" (UID: "6626f409-46a9-4003-85dc-3d6c53059674"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:01:38.010795 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:38.010770    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6626f409-46a9-4003-85dc-3d6c53059674-kube-api-access-jg7dp" pod "6626f409-46a9-4003-85dc-3d6c53059674" (UID: "6626f409-46a9-4003-85dc-3d6c53059674"). InnerVolumeSpecName "kube-api-access-jg7dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:01:38.013910 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:38.013844    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6626f409-46a9-4003-85dc-3d6c53059674-util" pod "6626f409-46a9-4003-85dc-3d6c53059674" (UID: "6626f409-46a9-4003-85dc-3d6c53059674"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:01:38.109543 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:38.109473    4464 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6626f409-46a9-4003-85dc-3d6c53059674-util\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:01:38.109543 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:38.109497    4464 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6626f409-46a9-4003-85dc-3d6c53059674-bundle\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:01:38.109543 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:38.109507    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg7dp\" (UniqueName: \"kubernetes.io/projected/6626f409-46a9-4003-85dc-3d6c53059674-kube-api-access-jg7dp\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:01:38.847405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:38.847372    4464 generic.go:356] "Generic (PLEG): container finished" podID="97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9" containerID="2d9e0f79976e0482a5d844917479ab2c9447134bf2f584fc67f7bc36fb7c6ab9" exitCode=0
Nov 02 12:01:38.847828 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:38.847457    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd" event={"ID":"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9","Type":"ContainerDied","Data":"2d9e0f79976e0482a5d844917479ab2c9447134bf2f584fc67f7bc36fb7c6ab9"}
Nov 02 12:01:38.849259 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:38.849237    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt" event={"ID":"6626f409-46a9-4003-85dc-3d6c53059674","Type":"ContainerDied","Data":"9268c73fbb204ae781703a2345119623c17c6fbeb4280743dc822bd24344e9d0"}
Nov 02 12:01:38.849361 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:38.849279    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9268c73fbb204ae781703a2345119623c17c6fbeb4280743dc822bd24344e9d0"
Nov 02 12:01:38.849361 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:38.849284    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lb4kt"
Nov 02 12:01:39.970005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:39.969965    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd"
Nov 02 12:01:40.024424 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:40.024397    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-bundle\") pod \"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9\" (UID: \"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9\") "
Nov 02 12:01:40.024565 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:40.024462    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-util\" (UniqueName: \"kubernetes.io/empty-dir/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-util\") pod \"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9\" (UID: \"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9\") "
Nov 02 12:01:40.024565 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:40.024515    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-kube-api-access-sqfjz\" (UniqueName: \"kubernetes.io/projected/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-kube-api-access-sqfjz\") pod \"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9\" (UID: \"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9\") "
Nov 02 12:01:40.025063 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:40.025037    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-bundle" pod "97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9" (UID: "97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:01:40.026576 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:40.026554    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-kube-api-access-sqfjz" pod "97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9" (UID: "97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9"). InnerVolumeSpecName "kube-api-access-sqfjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:01:40.028576 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:40.028557    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-util" pod "97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9" (UID: "97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:01:40.125518 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:40.125454    4464 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-util\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:01:40.125518 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:40.125479    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqfjz\" (UniqueName: \"kubernetes.io/projected/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-kube-api-access-sqfjz\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:01:40.125518 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:40.125491    4464 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9-bundle\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:01:40.862865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:40.862834    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd"
Nov 02 12:01:40.863066 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:40.862833    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj2vwd" event={"ID":"97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9","Type":"ContainerDied","Data":"e1aa305d9a564a2f53d23a190cb3ce95c2454288e87afb56fae5e16e14431000"}
Nov 02 12:01:40.863066 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:01:40.863008    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1aa305d9a564a2f53d23a190cb3ce95c2454288e87afb56fae5e16e14431000"
Nov 02 12:01:58.291803 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:01:58.291764    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:02:02.769505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.769475    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"]
Nov 02 12:02:02.769865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.769751    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6626f409-46a9-4003-85dc-3d6c53059674" containerName="util"
Nov 02 12:02:02.769865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.769761    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6626f409-46a9-4003-85dc-3d6c53059674" containerName="util"
Nov 02 12:02:02.769865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.769774    4464 cpu_manager.go:414] "containerMap: removing container" podUID="97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9" containerName="pull"
Nov 02 12:02:02.769865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.769778    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9" containerName="pull"
Nov 02 12:02:02.769865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.769785    4464 cpu_manager.go:414] "containerMap: removing container" podUID="97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9" containerName="extract"
Nov 02 12:02:02.769865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.769789    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9" containerName="extract"
Nov 02 12:02:02.769865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.769796    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6626f409-46a9-4003-85dc-3d6c53059674" containerName="pull"
Nov 02 12:02:02.769865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.769801    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6626f409-46a9-4003-85dc-3d6c53059674" containerName="pull"
Nov 02 12:02:02.769865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.769819    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6626f409-46a9-4003-85dc-3d6c53059674" containerName="extract"
Nov 02 12:02:02.769865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.769823    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6626f409-46a9-4003-85dc-3d6c53059674" containerName="extract"
Nov 02 12:02:02.769865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.769835    4464 cpu_manager.go:414] "containerMap: removing container" podUID="97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9" containerName="util"
Nov 02 12:02:02.769865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.769839    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9" containerName="util"
Nov 02 12:02:02.770246 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.769889    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a2d0dd-eede-4f2e-8ff5-2a03c70d74a9" containerName="extract"
Nov 02 12:02:02.770246 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.769898    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="6626f409-46a9-4003-85dc-3d6c53059674" containerName="extract"
Nov 02 12:02:02.772722 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.772704    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:02.877571 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.877537    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 12:02:02.877833 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.877655    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 12:02:02.877955 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.877934    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 12:02:02.878089 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.878036    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 12:02:02.878211 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.878196    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 12:02:02.878272 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.878241    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-qclbq\""
Nov 02 12:02:02.895238 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.895212    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"]
Nov 02 12:02:02.908081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.908051    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/53c36e06-31bf-4418-883f-a6800320143f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:02.908188 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.908087    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53c36e06-31bf-4418-883f-a6800320143f-webhook-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:02.908246 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.908224    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/53c36e06-31bf-4418-883f-a6800320143f-metrics-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:02.908285 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.908259    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhhl7\" (UniqueName: \"kubernetes.io/projected/53c36e06-31bf-4418-883f-a6800320143f-kube-api-access-lhhl7\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:02.908351 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.908330    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53c36e06-31bf-4418-883f-a6800320143f-tmp\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:02.908398 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:02.908370    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/53c36e06-31bf-4418-883f-a6800320143f-manifests\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:03.009451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:03.009420    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/53c36e06-31bf-4418-883f-a6800320143f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:03.009451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:03.009453    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53c36e06-31bf-4418-883f-a6800320143f-webhook-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:03.009640 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:03.009494    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/53c36e06-31bf-4418-883f-a6800320143f-metrics-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:03.009640 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:03.009515    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhhl7\" (UniqueName: \"kubernetes.io/projected/53c36e06-31bf-4418-883f-a6800320143f-kube-api-access-lhhl7\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:03.009640 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:03.009552    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53c36e06-31bf-4418-883f-a6800320143f-tmp\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:03.009640 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:03.009583    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/53c36e06-31bf-4418-883f-a6800320143f-manifests\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:03.010078 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:03.010045    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/53c36e06-31bf-4418-883f-a6800320143f-manifests\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:03.011926 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:03.011895    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/53c36e06-31bf-4418-883f-a6800320143f-tmp\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:03.012054 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:03.012016    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53c36e06-31bf-4418-883f-a6800320143f-webhook-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:03.012188 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:03.012164    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/53c36e06-31bf-4418-883f-a6800320143f-metrics-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:03.012227 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:03.012209    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/53c36e06-31bf-4418-883f-a6800320143f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:03.081694 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:03.081634    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhhl7\" (UniqueName: \"kubernetes.io/projected/53c36e06-31bf-4418-883f-a6800320143f-kube-api-access-lhhl7\") pod \"opendatahub-operator-controller-manager-9776d789f-nx47h\" (UID: \"53c36e06-31bf-4418-883f-a6800320143f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:03.086369 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:03.086287    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:03.577712 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:02:03.577685    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c36e06_31bf_4418_883f_a6800320143f.slice/crio-09bd8ac1ec4f7a1f96213f3b7ee7af750338efbb0a80d74362ca22f3283f5ab5 WatchSource:0}: Error finding container 09bd8ac1ec4f7a1f96213f3b7ee7af750338efbb0a80d74362ca22f3283f5ab5: Status 404 returned error can't find the container with id 09bd8ac1ec4f7a1f96213f3b7ee7af750338efbb0a80d74362ca22f3283f5ab5
Nov 02 12:02:03.581164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:03.581143    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"]
Nov 02 12:02:03.947815 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:03.947728    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h" event={"ID":"53c36e06-31bf-4418-883f-a6800320143f","Type":"ContainerStarted","Data":"09bd8ac1ec4f7a1f96213f3b7ee7af750338efbb0a80d74362ca22f3283f5ab5"}
Nov 02 12:02:07.967869 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:07.967791    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h" event={"ID":"53c36e06-31bf-4418-883f-a6800320143f","Type":"ContainerStarted","Data":"808a1b5989d584e5524941684e22d1b6b765bcc924f032da271ddc42e13dd3db"}
Nov 02 12:02:08.974796 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:08.974758    4464 generic.go:356] "Generic (PLEG): container finished" podID="53c36e06-31bf-4418-883f-a6800320143f" containerID="808a1b5989d584e5524941684e22d1b6b765bcc924f032da271ddc42e13dd3db" exitCode=0
Nov 02 12:02:08.975477 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:08.974828    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h" event={"ID":"53c36e06-31bf-4418-883f-a6800320143f","Type":"ContainerDied","Data":"808a1b5989d584e5524941684e22d1b6b765bcc924f032da271ddc42e13dd3db"}
Nov 02 12:02:09.980348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:09.980314    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h" event={"ID":"53c36e06-31bf-4418-883f-a6800320143f","Type":"ContainerStarted","Data":"2944b4b681ec888139d01df70eceab2b5e347ccd588c062313e06ce223d2fa06"}
Nov 02 12:02:09.980750 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:09.980443    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:10.225673 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:10.225626    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h" podStartSLOduration=2.84605601 podStartE2EDuration="8.225611864s" totalImagesPullingTime="4.950317248s" totalInitContainerRuntime="429.238606ms" isStatefulPod=false podCreationTimestamp="2026-11-02 12:02:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:02:10.174313427 +0000 UTC m=+358.738021872" watchObservedRunningTime="2026-11-02 12:02:10.225611864 +0000 UTC m=+358.789320271"
Nov 02 12:02:20.988871 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:20.988834    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-nx47h"
Nov 02 12:02:28.291361 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:02:28.291313    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:02:41.540492 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:41.540456    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85ccdd577f-4jv5l"]
Nov 02 12:02:41.548552 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:41.548519    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85ccdd577f-4jv5l"
Nov 02 12:02:41.601366 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:41.601332    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85ccdd577f-4jv5l"]
Nov 02 12:02:41.617444 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:41.617416    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a85197ff-7b08-4de8-967a-d0f3a381b53b-tmp\") pod \"kube-auth-proxy-85ccdd577f-4jv5l\" (UID: \"a85197ff-7b08-4de8-967a-d0f3a381b53b\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-4jv5l"
Nov 02 12:02:41.617575 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:41.617484    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85197ff-7b08-4de8-967a-d0f3a381b53b-tls-certs\") pod \"kube-auth-proxy-85ccdd577f-4jv5l\" (UID: \"a85197ff-7b08-4de8-967a-d0f3a381b53b\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-4jv5l"
Nov 02 12:02:41.617575 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:41.617554    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk9l5\" (UniqueName: \"kubernetes.io/projected/a85197ff-7b08-4de8-967a-d0f3a381b53b-kube-api-access-vk9l5\") pod \"kube-auth-proxy-85ccdd577f-4jv5l\" (UID: \"a85197ff-7b08-4de8-967a-d0f3a381b53b\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-4jv5l"
Nov 02 12:02:41.626168 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:41.626148    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 12:02:41.650492 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:41.650471    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-lw8pv\""
Nov 02 12:02:41.650589 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:41.650574    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 12:02:41.718172 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:41.718139    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85197ff-7b08-4de8-967a-d0f3a381b53b-tls-certs\") pod \"kube-auth-proxy-85ccdd577f-4jv5l\" (UID: \"a85197ff-7b08-4de8-967a-d0f3a381b53b\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-4jv5l"
Nov 02 12:02:41.718318 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:41.718184    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk9l5\" (UniqueName: \"kubernetes.io/projected/a85197ff-7b08-4de8-967a-d0f3a381b53b-kube-api-access-vk9l5\") pod \"kube-auth-proxy-85ccdd577f-4jv5l\" (UID: \"a85197ff-7b08-4de8-967a-d0f3a381b53b\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-4jv5l"
Nov 02 12:02:41.718318 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:41.718257    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a85197ff-7b08-4de8-967a-d0f3a381b53b-tmp\") pod \"kube-auth-proxy-85ccdd577f-4jv5l\" (UID: \"a85197ff-7b08-4de8-967a-d0f3a381b53b\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-4jv5l"
Nov 02 12:02:41.720867 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:41.720842    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85197ff-7b08-4de8-967a-d0f3a381b53b-tls-certs\") pod \"kube-auth-proxy-85ccdd577f-4jv5l\" (UID: \"a85197ff-7b08-4de8-967a-d0f3a381b53b\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-4jv5l"
Nov 02 12:02:41.721085 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:41.721064    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a85197ff-7b08-4de8-967a-d0f3a381b53b-tmp\") pod \"kube-auth-proxy-85ccdd577f-4jv5l\" (UID: \"a85197ff-7b08-4de8-967a-d0f3a381b53b\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-4jv5l"
Nov 02 12:02:41.782453 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:41.782425    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk9l5\" (UniqueName: \"kubernetes.io/projected/a85197ff-7b08-4de8-967a-d0f3a381b53b-kube-api-access-vk9l5\") pod \"kube-auth-proxy-85ccdd577f-4jv5l\" (UID: \"a85197ff-7b08-4de8-967a-d0f3a381b53b\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-4jv5l"
Nov 02 12:02:41.858742 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:41.858673    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85ccdd577f-4jv5l"
Nov 02 12:02:42.243809 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:42.243726    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85ccdd577f-4jv5l"]
Nov 02 12:02:42.246254 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:02:42.246227    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85197ff_7b08_4de8_967a_d0f3a381b53b.slice/crio-9b6d6d2297aed4958cbfea895221068419952b8d27d09d327233c47aee1bb32b WatchSource:0}: Error finding container 9b6d6d2297aed4958cbfea895221068419952b8d27d09d327233c47aee1bb32b: Status 404 returned error can't find the container with id 9b6d6d2297aed4958cbfea895221068419952b8d27d09d327233c47aee1bb32b
Nov 02 12:02:43.111869 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:43.111831    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85ccdd577f-4jv5l" event={"ID":"a85197ff-7b08-4de8-967a-d0f3a381b53b","Type":"ContainerStarted","Data":"9b6d6d2297aed4958cbfea895221068419952b8d27d09d327233c47aee1bb32b"}
Nov 02 12:02:45.122685 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:45.122653    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85ccdd577f-4jv5l" event={"ID":"a85197ff-7b08-4de8-967a-d0f3a381b53b","Type":"ContainerStarted","Data":"a7a0906d10e02e869c735eda4b6968fdf15038d77c15d374be9b3568ae0744ff"}
Nov 02 12:02:45.241805 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:45.241755    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85ccdd577f-4jv5l" podStartSLOduration=1.465656503 podStartE2EDuration="4.241740859s" totalImagesPullingTime="2.776084356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:02:45.240602297 +0000 UTC m=+393.804310706" watchObservedRunningTime="2026-11-02 12:02:45.241740859 +0000 UTC m=+393.805449300"
Nov 02 12:02:45.514414 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:45.514385    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:02:45.516564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:45.516543    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"
Nov 02 12:02:45.572048 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:45.572024    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 02 12:02:45.572170 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:45.572062    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 12:02:45.572224 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:45.572211    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 12:02:45.581198 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:45.581174    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:02:45.621537 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:45.621516    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-98q7s\""
Nov 02 12:02:45.655892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:45.655866    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41-config-file\") pod \"limitador-limitador-7c9b87d574-v6wvs\" (UID: \"89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"
Nov 02 12:02:45.656112 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:45.656087    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xdwc\" (UniqueName: \"kubernetes.io/projected/89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41-kube-api-access-2xdwc\") pod \"limitador-limitador-7c9b87d574-v6wvs\" (UID: \"89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"
Nov 02 12:02:45.757234 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:45.757204    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xdwc\" (UniqueName: \"kubernetes.io/projected/89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41-kube-api-access-2xdwc\") pod \"limitador-limitador-7c9b87d574-v6wvs\" (UID: \"89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"
Nov 02 12:02:45.757369 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:45.757275    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41-config-file\") pod \"limitador-limitador-7c9b87d574-v6wvs\" (UID: \"89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"
Nov 02 12:02:45.757805 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:45.757787    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41-config-file\") pod \"limitador-limitador-7c9b87d574-v6wvs\" (UID: \"89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"
Nov 02 12:02:45.822190 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:45.822136    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xdwc\" (UniqueName: \"kubernetes.io/projected/89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41-kube-api-access-2xdwc\") pod \"limitador-limitador-7c9b87d574-v6wvs\" (UID: \"89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"
Nov 02 12:02:45.827924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:45.827899    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"
Nov 02 12:02:45.830870 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:45.830850    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:02:46.228564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:46.228538    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:02:46.231681 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:02:46.231654    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ab4a35_50b7_46a3_b5a1_6f43ed3b9a41.slice/crio-7cf410b43ca57e5978dfd00a0283a571b82cb86d0aa3cb005adc24d4e5fb6a24 WatchSource:0}: Error finding container 7cf410b43ca57e5978dfd00a0283a571b82cb86d0aa3cb005adc24d4e5fb6a24: Status 404 returned error can't find the container with id 7cf410b43ca57e5978dfd00a0283a571b82cb86d0aa3cb005adc24d4e5fb6a24
Nov 02 12:02:47.132611 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:47.132577    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-v6wvs" event={"ID":"89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41","Type":"ContainerStarted","Data":"7cf410b43ca57e5978dfd00a0283a571b82cb86d0aa3cb005adc24d4e5fb6a24"}
Nov 02 12:02:50.151635 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:50.151590    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-v6wvs" event={"ID":"89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41","Type":"ContainerStarted","Data":"5e5a041544b0cc93bf3ca741616a9fdaabb796acf2d83f4576b91766290bed75"}
Nov 02 12:02:50.152032 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:50.151664    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"
Nov 02 12:02:50.278050 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:02:50.278000    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-v6wvs" podStartSLOduration=2.387740924 podStartE2EDuration="5.277970549s" totalImagesPullingTime="2.890229625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:02:50.276952416 +0000 UTC m=+398.840660826" watchObservedRunningTime="2026-11-02 12:02:50.277970549 +0000 UTC m=+398.841678982"
Nov 02 12:02:58.290863 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:02:58.290784    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:03:01.156875 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:01.156846    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"
Nov 02 12:03:11.588575 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:11.588547    4464 scope.go:122] "RemoveContainer" containerID="491d92ca92d6f67a078b6d7d3338af189b202ee9658f50af9acd79c2813ded58"
Nov 02 12:03:28.291234 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:03:28.291192    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:03:28.849304 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:28.849274    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65f5678545-r9w4j"]
Nov 02 12:03:28.851428 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:28.851408    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5678545-r9w4j"
Nov 02 12:03:28.899827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:28.899800    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2336892c-7add-4055-97ee-a0587546d3e8-webhook-cert\") pod \"maas-controller-65f5678545-r9w4j\" (UID: \"2336892c-7add-4055-97ee-a0587546d3e8\") " pod="opendatahub/maas-controller-65f5678545-r9w4j"
Nov 02 12:03:28.899942 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:28.899840    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2336892c-7add-4055-97ee-a0587546d3e8-metrics-certs\") pod \"maas-controller-65f5678545-r9w4j\" (UID: \"2336892c-7add-4055-97ee-a0587546d3e8\") " pod="opendatahub/maas-controller-65f5678545-r9w4j"
Nov 02 12:03:28.899942 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:28.899902    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmklc\" (UniqueName: \"kubernetes.io/projected/2336892c-7add-4055-97ee-a0587546d3e8-kube-api-access-xmklc\") pod \"maas-controller-65f5678545-r9w4j\" (UID: \"2336892c-7add-4055-97ee-a0587546d3e8\") " pod="opendatahub/maas-controller-65f5678545-r9w4j"
Nov 02 12:03:28.905544 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:28.905515    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 12:03:28.905686 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:28.905666    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 12:03:28.905768 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:28.905750    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 12:03:28.956496 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:28.956467    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 12:03:28.956682 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:28.956603    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 12:03:28.956768 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:28.956750    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tzzlq\""
Nov 02 12:03:28.958081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:28.958059    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f5678545-r9w4j"]
Nov 02 12:03:29.001321 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.001292    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2336892c-7add-4055-97ee-a0587546d3e8-webhook-cert\") pod \"maas-controller-65f5678545-r9w4j\" (UID: \"2336892c-7add-4055-97ee-a0587546d3e8\") " pod="opendatahub/maas-controller-65f5678545-r9w4j"
Nov 02 12:03:29.001459 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.001329    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2336892c-7add-4055-97ee-a0587546d3e8-metrics-certs\") pod \"maas-controller-65f5678545-r9w4j\" (UID: \"2336892c-7add-4055-97ee-a0587546d3e8\") " pod="opendatahub/maas-controller-65f5678545-r9w4j"
Nov 02 12:03:29.001459 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.001353    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmklc\" (UniqueName: \"kubernetes.io/projected/2336892c-7add-4055-97ee-a0587546d3e8-kube-api-access-xmklc\") pod \"maas-controller-65f5678545-r9w4j\" (UID: \"2336892c-7add-4055-97ee-a0587546d3e8\") " pod="opendatahub/maas-controller-65f5678545-r9w4j"
Nov 02 12:03:29.001459 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:03:29.001436    4464 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Nov 02 12:03:29.001572 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:03:29.001500    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2336892c-7add-4055-97ee-a0587546d3e8-webhook-cert podName:2336892c-7add-4055-97ee-a0587546d3e8 nodeName:}" failed. No retries permitted until 2026-11-02 12:03:29.501483284 +0000 UTC m=+438.065191674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2336892c-7add-4055-97ee-a0587546d3e8-webhook-cert") pod "maas-controller-65f5678545-r9w4j" (UID: "2336892c-7add-4055-97ee-a0587546d3e8") : secret "maas-controller-webhook-cert" not found
Nov 02 12:03:29.003744 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.003719    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2336892c-7add-4055-97ee-a0587546d3e8-metrics-certs\") pod \"maas-controller-65f5678545-r9w4j\" (UID: \"2336892c-7add-4055-97ee-a0587546d3e8\") " pod="opendatahub/maas-controller-65f5678545-r9w4j"
Nov 02 12:03:29.063585 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.063549    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmklc\" (UniqueName: \"kubernetes.io/projected/2336892c-7add-4055-97ee-a0587546d3e8-kube-api-access-xmklc\") pod \"maas-controller-65f5678545-r9w4j\" (UID: \"2336892c-7add-4055-97ee-a0587546d3e8\") " pod="opendatahub/maas-controller-65f5678545-r9w4j"
Nov 02 12:03:29.405553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.405515    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5975b94bc-pp42k"]
Nov 02 12:03:29.408043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.408020    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 12:03:29.469314 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.469285    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5975b94bc-pp42k"]
Nov 02 12:03:29.506353 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.506320    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2336892c-7add-4055-97ee-a0587546d3e8-webhook-cert\") pod \"maas-controller-65f5678545-r9w4j\" (UID: \"2336892c-7add-4055-97ee-a0587546d3e8\") " pod="opendatahub/maas-controller-65f5678545-r9w4j"
Nov 02 12:03:29.506353 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.506354    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-webhook-cert\") pod \"maas-controller-5975b94bc-pp42k\" (UID: \"3e629e5b-98b5-4987-83a2-dc5c8dfcb874\") " pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 12:03:29.506819 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.506396    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-metrics-certs\") pod \"maas-controller-5975b94bc-pp42k\" (UID: \"3e629e5b-98b5-4987-83a2-dc5c8dfcb874\") " pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 12:03:29.506819 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.506550    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hklq\" (UniqueName: \"kubernetes.io/projected/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-kube-api-access-5hklq\") pod \"maas-controller-5975b94bc-pp42k\" (UID: \"3e629e5b-98b5-4987-83a2-dc5c8dfcb874\") " pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 12:03:29.508738 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.508717    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2336892c-7add-4055-97ee-a0587546d3e8-webhook-cert\") pod \"maas-controller-65f5678545-r9w4j\" (UID: \"2336892c-7add-4055-97ee-a0587546d3e8\") " pod="opendatahub/maas-controller-65f5678545-r9w4j"
Nov 02 12:03:29.607609 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.607580    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-metrics-certs\") pod \"maas-controller-5975b94bc-pp42k\" (UID: \"3e629e5b-98b5-4987-83a2-dc5c8dfcb874\") " pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 12:03:29.607808 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.607620    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hklq\" (UniqueName: \"kubernetes.io/projected/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-kube-api-access-5hklq\") pod \"maas-controller-5975b94bc-pp42k\" (UID: \"3e629e5b-98b5-4987-83a2-dc5c8dfcb874\") " pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 12:03:29.607887 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.607802    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-webhook-cert\") pod \"maas-controller-5975b94bc-pp42k\" (UID: \"3e629e5b-98b5-4987-83a2-dc5c8dfcb874\") " pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 12:03:29.610075 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.610048    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-metrics-certs\") pod \"maas-controller-5975b94bc-pp42k\" (UID: \"3e629e5b-98b5-4987-83a2-dc5c8dfcb874\") " pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 12:03:29.610075 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.610060    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-webhook-cert\") pod \"maas-controller-5975b94bc-pp42k\" (UID: \"3e629e5b-98b5-4987-83a2-dc5c8dfcb874\") " pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 12:03:29.669564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.669501    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hklq\" (UniqueName: \"kubernetes.io/projected/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-kube-api-access-5hklq\") pod \"maas-controller-5975b94bc-pp42k\" (UID: \"3e629e5b-98b5-4987-83a2-dc5c8dfcb874\") " pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 12:03:29.720285 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.720257    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 12:03:29.764367 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:29.764334    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5678545-r9w4j"
Nov 02 12:03:30.144108 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:30.144080    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5975b94bc-pp42k"]
Nov 02 12:03:30.146218 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:03:30.146194    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e629e5b_98b5_4987_83a2_dc5c8dfcb874.slice/crio-052c87f4d2dae13cf5ed7829e6a85ff6177bcec0bd2dfaf67b80edb53e5ec493 WatchSource:0}: Error finding container 052c87f4d2dae13cf5ed7829e6a85ff6177bcec0bd2dfaf67b80edb53e5ec493: Status 404 returned error can't find the container with id 052c87f4d2dae13cf5ed7829e6a85ff6177bcec0bd2dfaf67b80edb53e5ec493
Nov 02 12:03:30.146605 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:30.146583    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65f5678545-r9w4j"]
Nov 02 12:03:30.186025 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:03:30.186000    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2336892c_7add_4055_97ee_a0587546d3e8.slice/crio-9a7a72a853335def0e70543ae97499928ad7e62bc3b9128d54d2f121afd9a51d WatchSource:0}: Error finding container 9a7a72a853335def0e70543ae97499928ad7e62bc3b9128d54d2f121afd9a51d: Status 404 returned error can't find the container with id 9a7a72a853335def0e70543ae97499928ad7e62bc3b9128d54d2f121afd9a51d
Nov 02 12:03:30.314134 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:30.314095    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5975b94bc-pp42k" event={"ID":"3e629e5b-98b5-4987-83a2-dc5c8dfcb874","Type":"ContainerStarted","Data":"052c87f4d2dae13cf5ed7829e6a85ff6177bcec0bd2dfaf67b80edb53e5ec493"}
Nov 02 12:03:30.315216 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:30.315190    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5678545-r9w4j" event={"ID":"2336892c-7add-4055-97ee-a0587546d3e8","Type":"ContainerStarted","Data":"9a7a72a853335def0e70543ae97499928ad7e62bc3b9128d54d2f121afd9a51d"}
Nov 02 12:03:37.347209 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:37.347165    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5678545-r9w4j" event={"ID":"2336892c-7add-4055-97ee-a0587546d3e8","Type":"ContainerStarted","Data":"41f64cee27d596b37d7ef54a07eaa65353e5e57a09cc54bae11e03b0f2ae8553"}
Nov 02 12:03:37.347690 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:37.347237    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65f5678545-r9w4j"
Nov 02 12:03:37.348589 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:37.348565    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5975b94bc-pp42k" event={"ID":"3e629e5b-98b5-4987-83a2-dc5c8dfcb874","Type":"ContainerStarted","Data":"2d2e9028093f07bfc6079e01c4b052b357fe115908a7f51f18bcb3c9f5422706"}
Nov 02 12:03:37.348707 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:37.348691    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 12:03:37.467218 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:37.467169    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-65f5678545-r9w4j" podStartSLOduration=3.131782368 podStartE2EDuration="9.467155802s" totalImagesPullingTime="6.335373434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:03:37.46611898 +0000 UTC m=+446.029827392" watchObservedRunningTime="2026-11-02 12:03:37.467155802 +0000 UTC m=+446.030864211"
Nov 02 12:03:37.587758 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:37.587697    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5975b94bc-pp42k" podStartSLOduration=2.203817938 podStartE2EDuration="8.587678419s" totalImagesPullingTime="6.383860481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:03:37.585592245 +0000 UTC m=+446.149300655" watchObservedRunningTime="2026-11-02 12:03:37.587678419 +0000 UTC m=+446.151386849"
Nov 02 12:03:40.011857 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.011826    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"]
Nov 02 12:03:40.015408 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.015392    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:03:40.068321 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.068300    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 02 12:03:40.068656 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.068633    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 02 12:03:40.094336 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.094310    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/98f4686d-b894-4e32-91bb-479a67c05b1d-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4z4sj\" (UID: \"98f4686d-b894-4e32-91bb-479a67c05b1d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:03:40.094457 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.094343    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/98f4686d-b894-4e32-91bb-479a67c05b1d-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4z4sj\" (UID: \"98f4686d-b894-4e32-91bb-479a67c05b1d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:03:40.094457 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.094371    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98f4686d-b894-4e32-91bb-479a67c05b1d-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4z4sj\" (UID: \"98f4686d-b894-4e32-91bb-479a67c05b1d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:03:40.094592 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.094567    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pnbp\" (UniqueName: \"kubernetes.io/projected/98f4686d-b894-4e32-91bb-479a67c05b1d-kube-api-access-2pnbp\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4z4sj\" (UID: \"98f4686d-b894-4e32-91bb-479a67c05b1d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:03:40.119597 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.119564    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 02 12:03:40.119597 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.119589    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-vcmmv\""
Nov 02 12:03:40.121241 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.121220    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"]
Nov 02 12:03:40.195329 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.195306    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pnbp\" (UniqueName: \"kubernetes.io/projected/98f4686d-b894-4e32-91bb-479a67c05b1d-kube-api-access-2pnbp\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4z4sj\" (UID: \"98f4686d-b894-4e32-91bb-479a67c05b1d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:03:40.195467 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.195346    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/98f4686d-b894-4e32-91bb-479a67c05b1d-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4z4sj\" (UID: \"98f4686d-b894-4e32-91bb-479a67c05b1d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:03:40.195467 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.195369    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/98f4686d-b894-4e32-91bb-479a67c05b1d-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4z4sj\" (UID: \"98f4686d-b894-4e32-91bb-479a67c05b1d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:03:40.195467 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.195393    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98f4686d-b894-4e32-91bb-479a67c05b1d-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4z4sj\" (UID: \"98f4686d-b894-4e32-91bb-479a67c05b1d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:03:40.195467 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:03:40.195427    4464 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 02 12:03:40.195467 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:03:40.195456    4464 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Nov 02 12:03:40.195710 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:03:40.195496    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/98f4686d-b894-4e32-91bb-479a67c05b1d-proxy-source podName:98f4686d-b894-4e32-91bb-479a67c05b1d nodeName:}" failed. No retries permitted until 2026-11-02 12:03:40.69548144 +0000 UTC m=+449.259189856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/98f4686d-b894-4e32-91bb-479a67c05b1d-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-4z4sj" (UID: "98f4686d-b894-4e32-91bb-479a67c05b1d") : configmap "usage-tenancy-proxy-source" not found
Nov 02 12:03:40.195710 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:03:40.195519    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98f4686d-b894-4e32-91bb-479a67c05b1d-tls-cert podName:98f4686d-b894-4e32-91bb-479a67c05b1d nodeName:}" failed. No retries permitted until 2026-11-02 12:03:40.695503577 +0000 UTC m=+449.259211964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/98f4686d-b894-4e32-91bb-479a67c05b1d-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-4z4sj" (UID: "98f4686d-b894-4e32-91bb-479a67c05b1d") : secret "usage-logs-tenancy-proxy-tls" not found
Nov 02 12:03:40.195944 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.195927    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98f4686d-b894-4e32-91bb-479a67c05b1d-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4z4sj\" (UID: \"98f4686d-b894-4e32-91bb-479a67c05b1d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:03:40.257824 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.257793    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pnbp\" (UniqueName: \"kubernetes.io/projected/98f4686d-b894-4e32-91bb-479a67c05b1d-kube-api-access-2pnbp\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4z4sj\" (UID: \"98f4686d-b894-4e32-91bb-479a67c05b1d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:03:40.698314 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.698276    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/98f4686d-b894-4e32-91bb-479a67c05b1d-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4z4sj\" (UID: \"98f4686d-b894-4e32-91bb-479a67c05b1d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:03:40.698314 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.698316    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/98f4686d-b894-4e32-91bb-479a67c05b1d-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4z4sj\" (UID: \"98f4686d-b894-4e32-91bb-479a67c05b1d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:03:40.699099 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.699075    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/98f4686d-b894-4e32-91bb-479a67c05b1d-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4z4sj\" (UID: \"98f4686d-b894-4e32-91bb-479a67c05b1d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:03:40.700734 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.700703    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/98f4686d-b894-4e32-91bb-479a67c05b1d-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4z4sj\" (UID: \"98f4686d-b894-4e32-91bb-479a67c05b1d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:03:40.927905 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:40.927870    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:03:41.298957 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:41.298932    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"]
Nov 02 12:03:41.301198 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:03:41.301169    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f4686d_b894_4e32_91bb_479a67c05b1d.slice/crio-535b34bfb9c2cec0933ada719711d4111afa0336b4af1c10fb0a28e4cceda664 WatchSource:0}: Error finding container 535b34bfb9c2cec0933ada719711d4111afa0336b4af1c10fb0a28e4cceda664: Status 404 returned error can't find the container with id 535b34bfb9c2cec0933ada719711d4111afa0336b4af1c10fb0a28e4cceda664
Nov 02 12:03:41.365676 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:41.365649    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj" event={"ID":"98f4686d-b894-4e32-91bb-479a67c05b1d","Type":"ContainerStarted","Data":"535b34bfb9c2cec0933ada719711d4111afa0336b4af1c10fb0a28e4cceda664"}
Nov 02 12:03:48.355192 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:48.354493    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65f5678545-r9w4j"
Nov 02 12:03:48.355192 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:48.354891    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 12:03:48.625933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:48.625862    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f5678545-r9w4j"]
Nov 02 12:03:48.626117 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:48.626092    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-65f5678545-r9w4j" podUID="2336892c-7add-4055-97ee-a0587546d3e8" containerName="manager" containerID="cri-o://41f64cee27d596b37d7ef54a07eaa65353e5e57a09cc54bae11e03b0f2ae8553" gracePeriod=30
Nov 02 12:03:49.013952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.013922    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5678545-r9w4j"
Nov 02 12:03:49.086228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.086196    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2336892c-7add-4055-97ee-a0587546d3e8-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2336892c-7add-4055-97ee-a0587546d3e8-webhook-cert\") pod \"2336892c-7add-4055-97ee-a0587546d3e8\" (UID: \"2336892c-7add-4055-97ee-a0587546d3e8\") "
Nov 02 12:03:49.086426 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.086243    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2336892c-7add-4055-97ee-a0587546d3e8-kube-api-access-xmklc\" (UniqueName: \"kubernetes.io/projected/2336892c-7add-4055-97ee-a0587546d3e8-kube-api-access-xmklc\") pod \"2336892c-7add-4055-97ee-a0587546d3e8\" (UID: \"2336892c-7add-4055-97ee-a0587546d3e8\") "
Nov 02 12:03:49.086426 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.086327    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2336892c-7add-4055-97ee-a0587546d3e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2336892c-7add-4055-97ee-a0587546d3e8-metrics-certs\") pod \"2336892c-7add-4055-97ee-a0587546d3e8\" (UID: \"2336892c-7add-4055-97ee-a0587546d3e8\") "
Nov 02 12:03:49.088887 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.088856    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2336892c-7add-4055-97ee-a0587546d3e8-metrics-certs" pod "2336892c-7add-4055-97ee-a0587546d3e8" (UID: "2336892c-7add-4055-97ee-a0587546d3e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:03:49.092323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.092287    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2336892c-7add-4055-97ee-a0587546d3e8-webhook-cert" pod "2336892c-7add-4055-97ee-a0587546d3e8" (UID: "2336892c-7add-4055-97ee-a0587546d3e8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:03:49.093887 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.093858    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2336892c-7add-4055-97ee-a0587546d3e8-kube-api-access-xmklc" pod "2336892c-7add-4055-97ee-a0587546d3e8" (UID: "2336892c-7add-4055-97ee-a0587546d3e8"). InnerVolumeSpecName "kube-api-access-xmklc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:03:49.187430 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.187359    4464 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2336892c-7add-4055-97ee-a0587546d3e8-webhook-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:03:49.187430 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.187394    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmklc\" (UniqueName: \"kubernetes.io/projected/2336892c-7add-4055-97ee-a0587546d3e8-kube-api-access-xmklc\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:03:49.187430 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.187410    4464 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2336892c-7add-4055-97ee-a0587546d3e8-metrics-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:03:49.404443 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.404391    4464 generic.go:356] "Generic (PLEG): container finished" podID="2336892c-7add-4055-97ee-a0587546d3e8" containerID="41f64cee27d596b37d7ef54a07eaa65353e5e57a09cc54bae11e03b0f2ae8553" exitCode=0
Nov 02 12:03:49.404858 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.404453    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65f5678545-r9w4j"
Nov 02 12:03:49.404858 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.404456    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5678545-r9w4j" event={"ID":"2336892c-7add-4055-97ee-a0587546d3e8","Type":"ContainerDied","Data":"41f64cee27d596b37d7ef54a07eaa65353e5e57a09cc54bae11e03b0f2ae8553"}
Nov 02 12:03:49.404858 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.404535    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65f5678545-r9w4j" event={"ID":"2336892c-7add-4055-97ee-a0587546d3e8","Type":"ContainerDied","Data":"9a7a72a853335def0e70543ae97499928ad7e62bc3b9128d54d2f121afd9a51d"}
Nov 02 12:03:49.404858 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.404564    4464 scope.go:122] "RemoveContainer" containerID="41f64cee27d596b37d7ef54a07eaa65353e5e57a09cc54bae11e03b0f2ae8553"
Nov 02 12:03:49.415242 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.415215    4464 scope.go:122] "RemoveContainer" containerID="41f64cee27d596b37d7ef54a07eaa65353e5e57a09cc54bae11e03b0f2ae8553"
Nov 02 12:03:49.415534 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:03:49.415515    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f64cee27d596b37d7ef54a07eaa65353e5e57a09cc54bae11e03b0f2ae8553\": container with ID starting with 41f64cee27d596b37d7ef54a07eaa65353e5e57a09cc54bae11e03b0f2ae8553 not found: ID does not exist" containerID="41f64cee27d596b37d7ef54a07eaa65353e5e57a09cc54bae11e03b0f2ae8553"
Nov 02 12:03:49.415601 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.415547    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f64cee27d596b37d7ef54a07eaa65353e5e57a09cc54bae11e03b0f2ae8553"} err="failed to get container status \"41f64cee27d596b37d7ef54a07eaa65353e5e57a09cc54bae11e03b0f2ae8553\": rpc error: code = NotFound desc = could not find container \"41f64cee27d596b37d7ef54a07eaa65353e5e57a09cc54bae11e03b0f2ae8553\": container with ID starting with 41f64cee27d596b37d7ef54a07eaa65353e5e57a09cc54bae11e03b0f2ae8553 not found: ID does not exist"
Nov 02 12:03:49.571548 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.571514    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65f5678545-r9w4j"]
Nov 02 12:03:49.622224 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:49.622186    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65f5678545-r9w4j"]
Nov 02 12:03:51.581903 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:51.581868    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2336892c-7add-4055-97ee-a0587546d3e8" path="/var/lib/kubelet/pods/2336892c-7add-4055-97ee-a0587546d3e8/volumes"
Nov 02 12:03:54.428961 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:54.428928    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj" event={"ID":"98f4686d-b894-4e32-91bb-479a67c05b1d","Type":"ContainerStarted","Data":"e79920a82d1a6ff0cce9aab74bf4944877d3c730e1f5463f1a2fb1493678e0e4"}
Nov 02 12:03:54.429377 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:54.429043    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:03:54.548138 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:03:54.548085    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj" podStartSLOduration=2.930507939 podStartE2EDuration="15.548069639s" totalImagesPullingTime="12.6175617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:03:54.547779074 +0000 UTC m=+463.111487482" watchObservedRunningTime="2026-11-02 12:03:54.548069639 +0000 UTC m=+463.111778050"
Nov 02 12:03:58.291140 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:03:58.291103    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:04:05.442696 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:05.442660    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4z4sj"
Nov 02 12:04:10.814200 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.814168    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"]
Nov 02 12:04:10.814646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.814540    4464 cpu_manager.go:414] "containerMap: removing container" podUID="2336892c-7add-4055-97ee-a0587546d3e8" containerName="manager"
Nov 02 12:04:10.814646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.814551    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2336892c-7add-4055-97ee-a0587546d3e8" containerName="manager"
Nov 02 12:04:10.814646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.814618    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="2336892c-7add-4055-97ee-a0587546d3e8" containerName="manager"
Nov 02 12:04:10.820919 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.820899    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.876405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.876372    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c036f63f-f1c7-4d49-be9e-9f8c60907554-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.876405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.876405    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c036f63f-f1c7-4d49-be9e-9f8c60907554-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.876610 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.876424    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c036f63f-f1c7-4d49-be9e-9f8c60907554-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.876610 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.876545    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c036f63f-f1c7-4d49-be9e-9f8c60907554-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.876610 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.876573    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c036f63f-f1c7-4d49-be9e-9f8c60907554-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.876703 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.876611    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9prmg\" (UniqueName: \"kubernetes.io/projected/c036f63f-f1c7-4d49-be9e-9f8c60907554-kube-api-access-9prmg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.876703 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.876680    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c036f63f-f1c7-4d49-be9e-9f8c60907554-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.926490 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.926459    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"]
Nov 02 12:04:10.977168 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.977135    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c036f63f-f1c7-4d49-be9e-9f8c60907554-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.977168 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.977168    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c036f63f-f1c7-4d49-be9e-9f8c60907554-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.977419 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.977201    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prmg\" (UniqueName: \"kubernetes.io/projected/c036f63f-f1c7-4d49-be9e-9f8c60907554-kube-api-access-9prmg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.977419 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.977333    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c036f63f-f1c7-4d49-be9e-9f8c60907554-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.977419 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.977394    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c036f63f-f1c7-4d49-be9e-9f8c60907554-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.977586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.977424    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c036f63f-f1c7-4d49-be9e-9f8c60907554-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.977586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.977456    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c036f63f-f1c7-4d49-be9e-9f8c60907554-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.978319 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.978289    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c036f63f-f1c7-4d49-be9e-9f8c60907554-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.979686 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.979662    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c036f63f-f1c7-4d49-be9e-9f8c60907554-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.979766 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.979729    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c036f63f-f1c7-4d49-be9e-9f8c60907554-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.979766 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.979747    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c036f63f-f1c7-4d49-be9e-9f8c60907554-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:10.979860 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:10.979838    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c036f63f-f1c7-4d49-be9e-9f8c60907554-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:11.038870 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:11.038835    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c036f63f-f1c7-4d49-be9e-9f8c60907554-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:11.039030 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:11.039005    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9prmg\" (UniqueName: \"kubernetes.io/projected/c036f63f-f1c7-4d49-be9e-9f8c60907554-kube-api-access-9prmg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-s2tz8\" (UID: \"c036f63f-f1c7-4d49-be9e-9f8c60907554\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:11.133084 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:11.133000    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:11.563883 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:11.563846    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"]
Nov 02 12:04:11.569700 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:04:11.569674    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc036f63f_f1c7_4d49_be9e_9f8c60907554.slice/crio-cf78626b4f667431e20b2620cb8c36803be5dbf4707c1503e4515bc37c330897 WatchSource:0}: Error finding container cf78626b4f667431e20b2620cb8c36803be5dbf4707c1503e4515bc37c330897: Status 404 returned error can't find the container with id cf78626b4f667431e20b2620cb8c36803be5dbf4707c1503e4515bc37c330897
Nov 02 12:04:11.571757 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:11.571727    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:04:11.571851 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:11.571784    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:04:12.505757 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:12.505720    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8" event={"ID":"c036f63f-f1c7-4d49-be9e-9f8c60907554","Type":"ContainerStarted","Data":"f1e83061f39b088106439b0f4835ce209dfcbcc77d435267b4e0db1ee38b7a9a"}
Nov 02 12:04:12.506277 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:12.505771    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8" event={"ID":"c036f63f-f1c7-4d49-be9e-9f8c60907554","Type":"ContainerStarted","Data":"cf78626b4f667431e20b2620cb8c36803be5dbf4707c1503e4515bc37c330897"}
Nov 02 12:04:12.506277 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:12.505899    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:12.507601 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:12.507578    4464 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-s2tz8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:04:12.507707 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:12.507621    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8" podUID="c036f63f-f1c7-4d49-be9e-9f8c60907554" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:04:12.661183 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:12.661129    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8" podStartSLOduration=2.661113471 podStartE2EDuration="2.661113471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:04:12.638150809 +0000 UTC m=+481.201859219" watchObservedRunningTime="2026-11-02 12:04:12.661113471 +0000 UTC m=+481.224821880"
Nov 02 12:04:13.511090 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:13.511063    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-s2tz8"
Nov 02 12:04:13.673437 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:13.673407    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"]
Nov 02 12:04:13.673608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:13.673588    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5" podUID="409d846e-dfe5-45c5-a672-d4bc9513d5b2" containerName="discovery" containerID="cri-o://b27e6ac3ecbc6ac814b34ad4cf55f9ed1783694a87d5b20d7f4a85a5afb41229" gracePeriod=30
Nov 02 12:04:13.779092 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:13.779058    4464 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-bwld5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.21:8080/ready\": dial tcp 10.133.0.21:8080: connect: connection refused" start-of-body=
Nov 02 12:04:13.779199 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:13.779111    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5" podUID="409d846e-dfe5-45c5-a672-d4bc9513d5b2" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.21:8080/ready\": dial tcp 10.133.0.21:8080: connect: connection refused"
Nov 02 12:04:13.915465 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:13.915443    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:04:14.009696 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.009667    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-kubeconfig\") pod \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") "
Nov 02 12:04:14.009881 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.009723    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-token\" (UniqueName: \"kubernetes.io/projected/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-token\") pod \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") "
Nov 02 12:04:14.009881 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.009743    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-csr-ca-configmap\") pod \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") "
Nov 02 12:04:14.009881 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.009779    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/409d846e-dfe5-45c5-a672-d4bc9513d5b2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/409d846e-dfe5-45c5-a672-d4bc9513d5b2-local-certs\") pod \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") "
Nov 02 12:04:14.009881 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.009808    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-cacerts\" (UniqueName: \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-cacerts\") pod \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") "
Nov 02 12:04:14.010113 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.009888    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-csr-dns-cert\") pod \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") "
Nov 02 12:04:14.010113 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.009964    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409d846e-dfe5-45c5-a672-d4bc9513d5b2-kube-api-access-zf5th\" (UniqueName: \"kubernetes.io/projected/409d846e-dfe5-45c5-a672-d4bc9513d5b2-kube-api-access-zf5th\") pod \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\" (UID: \"409d846e-dfe5-45c5-a672-d4bc9513d5b2\") "
Nov 02 12:04:14.010343 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.010299    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-csr-ca-configmap" pod "409d846e-dfe5-45c5-a672-d4bc9513d5b2" (UID: "409d846e-dfe5-45c5-a672-d4bc9513d5b2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:04:14.012333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.012309    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-kubeconfig" pod "409d846e-dfe5-45c5-a672-d4bc9513d5b2" (UID: "409d846e-dfe5-45c5-a672-d4bc9513d5b2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:04:14.012469 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.012337    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409d846e-dfe5-45c5-a672-d4bc9513d5b2-local-certs" pod "409d846e-dfe5-45c5-a672-d4bc9513d5b2" (UID: "409d846e-dfe5-45c5-a672-d4bc9513d5b2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:04:14.012535 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.012478    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409d846e-dfe5-45c5-a672-d4bc9513d5b2-kube-api-access-zf5th" pod "409d846e-dfe5-45c5-a672-d4bc9513d5b2" (UID: "409d846e-dfe5-45c5-a672-d4bc9513d5b2"). InnerVolumeSpecName "kube-api-access-zf5th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:04:14.012535 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.012479    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-token" pod "409d846e-dfe5-45c5-a672-d4bc9513d5b2" (UID: "409d846e-dfe5-45c5-a672-d4bc9513d5b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:04:14.012643 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.012552    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-cacerts" pod "409d846e-dfe5-45c5-a672-d4bc9513d5b2" (UID: "409d846e-dfe5-45c5-a672-d4bc9513d5b2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:04:14.012643 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.012564    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-csr-dns-cert" pod "409d846e-dfe5-45c5-a672-d4bc9513d5b2" (UID: "409d846e-dfe5-45c5-a672-d4bc9513d5b2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:04:14.111329 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.111295    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:04:14.111482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.111345    4464 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-csr-ca-configmap\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:04:14.111482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.111356    4464 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/409d846e-dfe5-45c5-a672-d4bc9513d5b2-local-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:04:14.111482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.111365    4464 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-cacerts\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:04:14.111482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.111375    4464 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-csr-dns-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:04:14.111482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.111383    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf5th\" (UniqueName: \"kubernetes.io/projected/409d846e-dfe5-45c5-a672-d4bc9513d5b2-kube-api-access-zf5th\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:04:14.111482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.111391    4464 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/409d846e-dfe5-45c5-a672-d4bc9513d5b2-istio-kubeconfig\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:04:14.515250 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.515216    4464 generic.go:356] "Generic (PLEG): container finished" podID="409d846e-dfe5-45c5-a672-d4bc9513d5b2" containerID="b27e6ac3ecbc6ac814b34ad4cf55f9ed1783694a87d5b20d7f4a85a5afb41229" exitCode=0
Nov 02 12:04:14.515650 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.515286    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"
Nov 02 12:04:14.515650 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.515302    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5" event={"ID":"409d846e-dfe5-45c5-a672-d4bc9513d5b2","Type":"ContainerDied","Data":"b27e6ac3ecbc6ac814b34ad4cf55f9ed1783694a87d5b20d7f4a85a5afb41229"}
Nov 02 12:04:14.515650 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.515384    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5" event={"ID":"409d846e-dfe5-45c5-a672-d4bc9513d5b2","Type":"ContainerDied","Data":"08c3c08b8970d10af03b759a66d07ec81f90e8d5bf60bbd0936c23d0af7055cc"}
Nov 02 12:04:14.515650 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.515409    4464 scope.go:122] "RemoveContainer" containerID="b27e6ac3ecbc6ac814b34ad4cf55f9ed1783694a87d5b20d7f4a85a5afb41229"
Nov 02 12:04:14.523785 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.523769    4464 scope.go:122] "RemoveContainer" containerID="b27e6ac3ecbc6ac814b34ad4cf55f9ed1783694a87d5b20d7f4a85a5afb41229"
Nov 02 12:04:14.524048 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:04:14.524032    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27e6ac3ecbc6ac814b34ad4cf55f9ed1783694a87d5b20d7f4a85a5afb41229\": container with ID starting with b27e6ac3ecbc6ac814b34ad4cf55f9ed1783694a87d5b20d7f4a85a5afb41229 not found: ID does not exist" containerID="b27e6ac3ecbc6ac814b34ad4cf55f9ed1783694a87d5b20d7f4a85a5afb41229"
Nov 02 12:04:14.524092 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.524057    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27e6ac3ecbc6ac814b34ad4cf55f9ed1783694a87d5b20d7f4a85a5afb41229"} err="failed to get container status \"b27e6ac3ecbc6ac814b34ad4cf55f9ed1783694a87d5b20d7f4a85a5afb41229\": rpc error: code = NotFound desc = could not find container \"b27e6ac3ecbc6ac814b34ad4cf55f9ed1783694a87d5b20d7f4a85a5afb41229\": container with ID starting with b27e6ac3ecbc6ac814b34ad4cf55f9ed1783694a87d5b20d7f4a85a5afb41229 not found: ID does not exist"
Nov 02 12:04:14.683273 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.683238    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"]
Nov 02 12:04:14.733565 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:14.733531    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-bwld5"]
Nov 02 12:04:15.580664 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:15.580629    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="409d846e-dfe5-45c5-a672-d4bc9513d5b2" path="/var/lib/kubelet/pods/409d846e-dfe5-45c5-a672-d4bc9513d5b2/volumes"
Nov 02 12:04:16.335025 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.334993    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-28669"]
Nov 02 12:04:16.335359 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.335347    4464 cpu_manager.go:414] "containerMap: removing container" podUID="409d846e-dfe5-45c5-a672-d4bc9513d5b2" containerName="discovery"
Nov 02 12:04:16.335405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.335360    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409d846e-dfe5-45c5-a672-d4bc9513d5b2" containerName="discovery"
Nov 02 12:04:16.335437 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.335429    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="409d846e-dfe5-45c5-a672-d4bc9513d5b2" containerName="discovery"
Nov 02 12:04:16.339629 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.339612    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-28669"
Nov 02 12:04:16.397814 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.397784    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-28669"]
Nov 02 12:04:16.431625 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.431588    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpj88\" (UniqueName: \"kubernetes.io/projected/5d927c4f-316f-441e-bf28-dfbae4d88bfd-kube-api-access-wpj88\") pod \"kserve-controller-manager-776cc66d47-28669\" (UID: \"5d927c4f-316f-441e-bf28-dfbae4d88bfd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-28669"
Nov 02 12:04:16.431764 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.431724    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d927c4f-316f-441e-bf28-dfbae4d88bfd-cert\") pod \"kserve-controller-manager-776cc66d47-28669\" (UID: \"5d927c4f-316f-441e-bf28-dfbae4d88bfd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-28669"
Nov 02 12:04:16.437509 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.437486    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 02 12:04:16.437708 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.437692    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-d46kt\""
Nov 02 12:04:16.532417 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.532384    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d927c4f-316f-441e-bf28-dfbae4d88bfd-cert\") pod \"kserve-controller-manager-776cc66d47-28669\" (UID: \"5d927c4f-316f-441e-bf28-dfbae4d88bfd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-28669"
Nov 02 12:04:16.532576 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.532452    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpj88\" (UniqueName: \"kubernetes.io/projected/5d927c4f-316f-441e-bf28-dfbae4d88bfd-kube-api-access-wpj88\") pod \"kserve-controller-manager-776cc66d47-28669\" (UID: \"5d927c4f-316f-441e-bf28-dfbae4d88bfd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-28669"
Nov 02 12:04:16.534844 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.534820    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d927c4f-316f-441e-bf28-dfbae4d88bfd-cert\") pod \"kserve-controller-manager-776cc66d47-28669\" (UID: \"5d927c4f-316f-441e-bf28-dfbae4d88bfd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-28669"
Nov 02 12:04:16.594326 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.594257    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpj88\" (UniqueName: \"kubernetes.io/projected/5d927c4f-316f-441e-bf28-dfbae4d88bfd-kube-api-access-wpj88\") pod \"kserve-controller-manager-776cc66d47-28669\" (UID: \"5d927c4f-316f-441e-bf28-dfbae4d88bfd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-28669"
Nov 02 12:04:16.651788 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.651754    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-28669"
Nov 02 12:04:16.682233 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.682204    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-6sbtx"]
Nov 02 12:04:16.685902 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.685879    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-6sbtx"
Nov 02 12:04:16.734639 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.734611    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c5948cf-a3f1-4f12-b4ac-073db02a1911-cert\") pod \"odh-model-controller-769875db56-6sbtx\" (UID: \"5c5948cf-a3f1-4f12-b4ac-073db02a1911\") " pod="opendatahub/odh-model-controller-769875db56-6sbtx"
Nov 02 12:04:16.734792 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.734647    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7w6b\" (UniqueName: \"kubernetes.io/projected/5c5948cf-a3f1-4f12-b4ac-073db02a1911-kube-api-access-c7w6b\") pod \"odh-model-controller-769875db56-6sbtx\" (UID: \"5c5948cf-a3f1-4f12-b4ac-073db02a1911\") " pod="opendatahub/odh-model-controller-769875db56-6sbtx"
Nov 02 12:04:16.734792 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.734669    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c5948cf-a3f1-4f12-b4ac-073db02a1911-metrics-certs\") pod \"odh-model-controller-769875db56-6sbtx\" (UID: \"5c5948cf-a3f1-4f12-b4ac-073db02a1911\") " pod="opendatahub/odh-model-controller-769875db56-6sbtx"
Nov 02 12:04:16.742094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.742072    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-lqw2p\""
Nov 02 12:04:16.744925 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.744901    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-6sbtx"]
Nov 02 12:04:16.766827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.766789    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 02 12:04:16.766946 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.766913    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 02 12:04:16.835273 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.835240    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c5948cf-a3f1-4f12-b4ac-073db02a1911-cert\") pod \"odh-model-controller-769875db56-6sbtx\" (UID: \"5c5948cf-a3f1-4f12-b4ac-073db02a1911\") " pod="opendatahub/odh-model-controller-769875db56-6sbtx"
Nov 02 12:04:16.835439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.835279    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7w6b\" (UniqueName: \"kubernetes.io/projected/5c5948cf-a3f1-4f12-b4ac-073db02a1911-kube-api-access-c7w6b\") pod \"odh-model-controller-769875db56-6sbtx\" (UID: \"5c5948cf-a3f1-4f12-b4ac-073db02a1911\") " pod="opendatahub/odh-model-controller-769875db56-6sbtx"
Nov 02 12:04:16.835439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.835303    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c5948cf-a3f1-4f12-b4ac-073db02a1911-metrics-certs\") pod \"odh-model-controller-769875db56-6sbtx\" (UID: \"5c5948cf-a3f1-4f12-b4ac-073db02a1911\") " pod="opendatahub/odh-model-controller-769875db56-6sbtx"
Nov 02 12:04:16.837763 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.837735    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c5948cf-a3f1-4f12-b4ac-073db02a1911-metrics-certs\") pod \"odh-model-controller-769875db56-6sbtx\" (UID: \"5c5948cf-a3f1-4f12-b4ac-073db02a1911\") " pod="opendatahub/odh-model-controller-769875db56-6sbtx"
Nov 02 12:04:16.837891 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.837749    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c5948cf-a3f1-4f12-b4ac-073db02a1911-cert\") pod \"odh-model-controller-769875db56-6sbtx\" (UID: \"5c5948cf-a3f1-4f12-b4ac-073db02a1911\") " pod="opendatahub/odh-model-controller-769875db56-6sbtx"
Nov 02 12:04:16.897070 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.896991    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7w6b\" (UniqueName: \"kubernetes.io/projected/5c5948cf-a3f1-4f12-b4ac-073db02a1911-kube-api-access-c7w6b\") pod \"odh-model-controller-769875db56-6sbtx\" (UID: \"5c5948cf-a3f1-4f12-b4ac-073db02a1911\") " pod="opendatahub/odh-model-controller-769875db56-6sbtx"
Nov 02 12:04:16.999307 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:16.999269    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-6sbtx"
Nov 02 12:04:17.021910 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:17.021888    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-28669"]
Nov 02 12:04:17.024078 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:04:17.024050    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d927c4f_316f_441e_bf28_dfbae4d88bfd.slice/crio-6638f718cbdb1340a13dff43ee050135ba3c28067d162bd157eb6bbb5e4de0d7 WatchSource:0}: Error finding container 6638f718cbdb1340a13dff43ee050135ba3c28067d162bd157eb6bbb5e4de0d7: Status 404 returned error can't find the container with id 6638f718cbdb1340a13dff43ee050135ba3c28067d162bd157eb6bbb5e4de0d7
Nov 02 12:04:17.368616 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:17.368460    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-6sbtx"]
Nov 02 12:04:17.370783 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:04:17.370756    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5948cf_a3f1_4f12_b4ac_073db02a1911.slice/crio-3b41d92192e6cd9bcda5859a24ec927db697ef4c28d294ea5514cacf4d696f4d WatchSource:0}: Error finding container 3b41d92192e6cd9bcda5859a24ec927db697ef4c28d294ea5514cacf4d696f4d: Status 404 returned error can't find the container with id 3b41d92192e6cd9bcda5859a24ec927db697ef4c28d294ea5514cacf4d696f4d
Nov 02 12:04:17.530628 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:17.530573    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-28669" event={"ID":"5d927c4f-316f-441e-bf28-dfbae4d88bfd","Type":"ContainerStarted","Data":"6638f718cbdb1340a13dff43ee050135ba3c28067d162bd157eb6bbb5e4de0d7"}
Nov 02 12:04:17.531614 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:17.531592    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-6sbtx" event={"ID":"5c5948cf-a3f1-4f12-b4ac-073db02a1911","Type":"ContainerStarted","Data":"3b41d92192e6cd9bcda5859a24ec927db697ef4c28d294ea5514cacf4d696f4d"}
Nov 02 12:04:20.551801 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:20.551762    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-28669" event={"ID":"5d927c4f-316f-441e-bf28-dfbae4d88bfd","Type":"ContainerStarted","Data":"fd68cdbea49f96ecb97b3ad4523754a823fb6cb87506cbc4ebc5880d6b28c47a"}
Nov 02 12:04:20.552224 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:20.551863    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-28669"
Nov 02 12:04:20.553597 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:20.553565    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-6sbtx" event={"ID":"5c5948cf-a3f1-4f12-b4ac-073db02a1911","Type":"ContainerStarted","Data":"fe7128fee4dfcb313a8aa28371bc5d9c10517ca77533862a2a4c9e8450461ae8"}
Nov 02 12:04:20.553747 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:20.553625    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-6sbtx"
Nov 02 12:04:20.670904 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:20.670839    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-28669" podStartSLOduration=2.215523409 podStartE2EDuration="4.670820358s" totalImagesPullingTime="2.455296949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:04:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:04:20.669925638 +0000 UTC m=+489.233634050" watchObservedRunningTime="2026-11-02 12:04:20.670820358 +0000 UTC m=+489.234528768"
Nov 02 12:04:20.790110 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:20.790059    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-6sbtx" podStartSLOduration=1.702727044 podStartE2EDuration="4.790044982s" totalImagesPullingTime="3.087317938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:04:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:04:20.789266593 +0000 UTC m=+489.352975002" watchObservedRunningTime="2026-11-02 12:04:20.790044982 +0000 UTC m=+489.353753391"
Nov 02 12:04:21.106423 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:21.106339    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:04:28.291609 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:04:28.291520    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:04:31.560352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:31.560323    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-6sbtx"
Nov 02 12:04:51.562600 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:04:51.562568    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-28669"
Nov 02 12:04:58.290525 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:04:58.290481    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:05:28.291716 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:05:28.291675    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:05:58.291554 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:05:58.291449    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:06:08.026656 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:08.026621    4464 generic.go:356] "Generic (PLEG): container finished" podID="3e629e5b-98b5-4987-83a2-dc5c8dfcb874" containerID="2d2e9028093f07bfc6079e01c4b052b357fe115908a7f51f18bcb3c9f5422706" exitCode=1
Nov 02 12:06:08.026656 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:08.026658    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5975b94bc-pp42k" event={"ID":"3e629e5b-98b5-4987-83a2-dc5c8dfcb874","Type":"ContainerDied","Data":"2d2e9028093f07bfc6079e01c4b052b357fe115908a7f51f18bcb3c9f5422706"}
Nov 02 12:06:08.027127 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:08.027005    4464 scope.go:122] "RemoveContainer" containerID="2d2e9028093f07bfc6079e01c4b052b357fe115908a7f51f18bcb3c9f5422706"
Nov 02 12:06:08.353057 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:08.353024    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 12:06:09.037773 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:09.037740    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5975b94bc-pp42k" event={"ID":"3e629e5b-98b5-4987-83a2-dc5c8dfcb874","Type":"ContainerStarted","Data":"c100f80b15aa3a0a70d574ff1134fe4bb1ae76f4d2439675cc223144cc09556f"}
Nov 02 12:06:09.038219 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:09.037898    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 12:06:09.438443 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:09.438410    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2xzcb"]
Nov 02 12:06:09.441131 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:09.441106    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb"
Nov 02 12:06:09.497125 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:09.497092    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-lmz6z\""
Nov 02 12:06:09.499023 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:09.498998    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2xzcb"]
Nov 02 12:06:09.499909 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:09.499886    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 02 12:06:09.500328 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:09.500307    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 12:06:09.601961 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:09.601929    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/334bc218-c5a6-41ac-bf0b-2818e2033c13-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-2xzcb\" (UID: \"334bc218-c5a6-41ac-bf0b-2818e2033c13\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb"
Nov 02 12:06:09.602097 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:09.602003    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mp4j\" (UniqueName: \"kubernetes.io/projected/334bc218-c5a6-41ac-bf0b-2818e2033c13-kube-api-access-4mp4j\") pod \"ai-gateway-controller-7d85675f98-2xzcb\" (UID: \"334bc218-c5a6-41ac-bf0b-2818e2033c13\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb"
Nov 02 12:06:09.703312 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:09.703244    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/334bc218-c5a6-41ac-bf0b-2818e2033c13-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-2xzcb\" (UID: \"334bc218-c5a6-41ac-bf0b-2818e2033c13\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb"
Nov 02 12:06:09.703312 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:09.703281    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mp4j\" (UniqueName: \"kubernetes.io/projected/334bc218-c5a6-41ac-bf0b-2818e2033c13-kube-api-access-4mp4j\") pod \"ai-gateway-controller-7d85675f98-2xzcb\" (UID: \"334bc218-c5a6-41ac-bf0b-2818e2033c13\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb"
Nov 02 12:06:09.705756 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:09.705730    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/334bc218-c5a6-41ac-bf0b-2818e2033c13-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-2xzcb\" (UID: \"334bc218-c5a6-41ac-bf0b-2818e2033c13\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb"
Nov 02 12:06:09.764802 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:09.764771    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mp4j\" (UniqueName: \"kubernetes.io/projected/334bc218-c5a6-41ac-bf0b-2818e2033c13-kube-api-access-4mp4j\") pod \"ai-gateway-controller-7d85675f98-2xzcb\" (UID: \"334bc218-c5a6-41ac-bf0b-2818e2033c13\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb"
Nov 02 12:06:10.057569 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:10.057537    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb"
Nov 02 12:06:10.428668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:10.428646    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2xzcb"]
Nov 02 12:06:10.430631 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:06:10.430601    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334bc218_c5a6_41ac_bf0b_2818e2033c13.slice/crio-04fc1112cb1afc29044427a97723de7a8291edd9024f012b7b5ba21e5ce223d3 WatchSource:0}: Error finding container 04fc1112cb1afc29044427a97723de7a8291edd9024f012b7b5ba21e5ce223d3: Status 404 returned error can't find the container with id 04fc1112cb1afc29044427a97723de7a8291edd9024f012b7b5ba21e5ce223d3
Nov 02 12:06:11.047673 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:11.047637    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb" event={"ID":"334bc218-c5a6-41ac-bf0b-2818e2033c13","Type":"ContainerStarted","Data":"04fc1112cb1afc29044427a97723de7a8291edd9024f012b7b5ba21e5ce223d3"}
Nov 02 12:06:11.616220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:11.616192    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:06:11.616220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:11.616210    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:06:13.493713 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:13.493682    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 12:06:14.064013 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:14.063959    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb" event={"ID":"334bc218-c5a6-41ac-bf0b-2818e2033c13","Type":"ContainerStarted","Data":"1aaf47510a13fd698670a80ae1139fd9aeb48ee30237cbc35a22e4e0a21e9a16"}
Nov 02 12:06:14.064212 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:14.064068    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb"
Nov 02 12:06:14.181912 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:14.181852    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb" podStartSLOduration=2.173178549 podStartE2EDuration="5.18184035s" totalImagesPullingTime="3.008661801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:06:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:06:14.181596206 +0000 UTC m=+602.745304607" watchObservedRunningTime="2026-11-02 12:06:14.18184035 +0000 UTC m=+602.745548759"
Nov 02 12:06:17.857951 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:17.857922    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"]
Nov 02 12:06:17.860415 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:17.860399    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"
Nov 02 12:06:17.913998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:17.913945    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 12:06:17.914143 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:17.914004    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-hzvfw\""
Nov 02 12:06:17.966093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:17.966063    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"]
Nov 02 12:06:17.976461 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:17.976432    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwc7d\" (UniqueName: \"kubernetes.io/projected/0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c-kube-api-access-rwc7d\") pod \"payload-pre-processing-7b98bb678c-hf7lc\" (UID: \"0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"
Nov 02 12:06:17.976573 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:17.976476    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-hf7lc\" (UID: \"0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"
Nov 02 12:06:17.992281 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:17.992255    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"]
Nov 02 12:06:17.994563 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:17.994548    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"
Nov 02 12:06:18.056332 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:18.056301    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"]
Nov 02 12:06:18.077017 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:18.076970    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-hf7lc\" (UID: \"0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"
Nov 02 12:06:18.077171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:18.077040    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gztzz\" (UniqueName: \"kubernetes.io/projected/5c2d3d77-9d5c-457e-b93b-c94cf304e188-kube-api-access-gztzz\") pod \"payload-processing-55fbbbf9dc-z7qfl\" (UID: \"5c2d3d77-9d5c-457e-b93b-c94cf304e188\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"
Nov 02 12:06:18.077171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:18.077071    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2d3d77-9d5c-457e-b93b-c94cf304e188-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-z7qfl\" (UID: \"5c2d3d77-9d5c-457e-b93b-c94cf304e188\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"
Nov 02 12:06:18.077171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:18.077119    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwc7d\" (UniqueName: \"kubernetes.io/projected/0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c-kube-api-access-rwc7d\") pod \"payload-pre-processing-7b98bb678c-hf7lc\" (UID: \"0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"
Nov 02 12:06:18.077714 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:18.077691    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-hf7lc\" (UID: \"0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"
Nov 02 12:06:18.140079 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:18.140000    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwc7d\" (UniqueName: \"kubernetes.io/projected/0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c-kube-api-access-rwc7d\") pod \"payload-pre-processing-7b98bb678c-hf7lc\" (UID: \"0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"
Nov 02 12:06:18.171612 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:18.171575    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"
Nov 02 12:06:18.178667 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:18.178644    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gztzz\" (UniqueName: \"kubernetes.io/projected/5c2d3d77-9d5c-457e-b93b-c94cf304e188-kube-api-access-gztzz\") pod \"payload-processing-55fbbbf9dc-z7qfl\" (UID: \"5c2d3d77-9d5c-457e-b93b-c94cf304e188\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"
Nov 02 12:06:18.178779 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:18.178680    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2d3d77-9d5c-457e-b93b-c94cf304e188-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-z7qfl\" (UID: \"5c2d3d77-9d5c-457e-b93b-c94cf304e188\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"
Nov 02 12:06:18.179295 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:18.179277    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2d3d77-9d5c-457e-b93b-c94cf304e188-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-z7qfl\" (UID: \"5c2d3d77-9d5c-457e-b93b-c94cf304e188\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"
Nov 02 12:06:18.240847 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:18.240817    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gztzz\" (UniqueName: \"kubernetes.io/projected/5c2d3d77-9d5c-457e-b93b-c94cf304e188-kube-api-access-gztzz\") pod \"payload-processing-55fbbbf9dc-z7qfl\" (UID: \"5c2d3d77-9d5c-457e-b93b-c94cf304e188\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"
Nov 02 12:06:18.306408 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:18.306374    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"
Nov 02 12:06:18.538781 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:18.538755    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"]
Nov 02 12:06:18.540788 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:06:18.540764    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc6178f_53db_4bf6_ae1f_3def4e3b9f4c.slice/crio-d786a8faaad3c431417c0a4df94a5393c39e888db8de4272b9970acc0c276420 WatchSource:0}: Error finding container d786a8faaad3c431417c0a4df94a5393c39e888db8de4272b9970acc0c276420: Status 404 returned error can't find the container with id d786a8faaad3c431417c0a4df94a5393c39e888db8de4272b9970acc0c276420
Nov 02 12:06:18.672551 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:18.672529    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"]
Nov 02 12:06:18.674562 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:06:18.674536    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2d3d77_9d5c_457e_b93b_c94cf304e188.slice/crio-7e8a7f3a0b542ac3d89169bd761a0511189ef38e8a6bed83aed29fbae38832a8 WatchSource:0}: Error finding container 7e8a7f3a0b542ac3d89169bd761a0511189ef38e8a6bed83aed29fbae38832a8: Status 404 returned error can't find the container with id 7e8a7f3a0b542ac3d89169bd761a0511189ef38e8a6bed83aed29fbae38832a8
Nov 02 12:06:18.676453 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:18.676436    4464 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:06:19.088128 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:19.088091    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl" event={"ID":"5c2d3d77-9d5c-457e-b93b-c94cf304e188","Type":"ContainerStarted","Data":"7e8a7f3a0b542ac3d89169bd761a0511189ef38e8a6bed83aed29fbae38832a8"}
Nov 02 12:06:19.089347 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:19.089323    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc" event={"ID":"0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c","Type":"ContainerStarted","Data":"d786a8faaad3c431417c0a4df94a5393c39e888db8de4272b9970acc0c276420"}
Nov 02 12:06:20.044833 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:20.044799    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 12:06:22.104686 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:22.104653    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl" event={"ID":"5c2d3d77-9d5c-457e-b93b-c94cf304e188","Type":"ContainerStarted","Data":"f36c41ef7fc35f2ad29a3f2b386a16d077e9a9f3dd4918191f6ec9eeceef00b6"}
Nov 02 12:06:22.105144 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:22.104800    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"
Nov 02 12:06:22.106159 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:22.106135    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc" event={"ID":"0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c","Type":"ContainerStarted","Data":"655ff2677b95df3b8230fe110c472ed137f27fb271f41f592adfaabebf3b776c"}
Nov 02 12:06:22.106272 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:22.106258    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"
Nov 02 12:06:22.223653 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:22.223612    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl" podStartSLOduration=2.558709367 podStartE2EDuration="5.223601714s" totalImagesPullingTime="2.664892347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:06:22.222730417 +0000 UTC m=+610.786438827" watchObservedRunningTime="2026-11-02 12:06:22.223601714 +0000 UTC m=+610.787310125"
Nov 02 12:06:22.342506 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:22.342459    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc" podStartSLOduration=2.603025222 podStartE2EDuration="5.34244433s" totalImagesPullingTime="2.739419108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:06:22.341554761 +0000 UTC m=+610.905263171" watchObservedRunningTime="2026-11-02 12:06:22.34244433 +0000 UTC m=+610.906152738"
Nov 02 12:06:25.070667 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:25.070636    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb"
Nov 02 12:06:25.430100 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:25.430003    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-dcd84bcbc-k6pdq"]
Nov 02 12:06:25.432483 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:25.432461    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-dcd84bcbc-k6pdq"
Nov 02 12:06:25.492549 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:25.492521    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-dcd84bcbc-k6pdq"]
Nov 02 12:06:25.538568 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:25.538532    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwnpv\" (UniqueName: \"kubernetes.io/projected/17f4db95-d03a-4200-8b01-d7760251e7fa-kube-api-access-fwnpv\") pod \"ai-gateway-controller-dcd84bcbc-k6pdq\" (UID: \"17f4db95-d03a-4200-8b01-d7760251e7fa\") " pod="opendatahub/ai-gateway-controller-dcd84bcbc-k6pdq"
Nov 02 12:06:25.538728 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:25.538600    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f4db95-d03a-4200-8b01-d7760251e7fa-metrics-certs\") pod \"ai-gateway-controller-dcd84bcbc-k6pdq\" (UID: \"17f4db95-d03a-4200-8b01-d7760251e7fa\") " pod="opendatahub/ai-gateway-controller-dcd84bcbc-k6pdq"
Nov 02 12:06:25.639875 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:25.639825    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f4db95-d03a-4200-8b01-d7760251e7fa-metrics-certs\") pod \"ai-gateway-controller-dcd84bcbc-k6pdq\" (UID: \"17f4db95-d03a-4200-8b01-d7760251e7fa\") " pod="opendatahub/ai-gateway-controller-dcd84bcbc-k6pdq"
Nov 02 12:06:25.640087 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:25.639931    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwnpv\" (UniqueName: \"kubernetes.io/projected/17f4db95-d03a-4200-8b01-d7760251e7fa-kube-api-access-fwnpv\") pod \"ai-gateway-controller-dcd84bcbc-k6pdq\" (UID: \"17f4db95-d03a-4200-8b01-d7760251e7fa\") " pod="opendatahub/ai-gateway-controller-dcd84bcbc-k6pdq"
Nov 02 12:06:25.642343 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:25.642312    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17f4db95-d03a-4200-8b01-d7760251e7fa-metrics-certs\") pod \"ai-gateway-controller-dcd84bcbc-k6pdq\" (UID: \"17f4db95-d03a-4200-8b01-d7760251e7fa\") " pod="opendatahub/ai-gateway-controller-dcd84bcbc-k6pdq"
Nov 02 12:06:25.702427 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:25.702358    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwnpv\" (UniqueName: \"kubernetes.io/projected/17f4db95-d03a-4200-8b01-d7760251e7fa-kube-api-access-fwnpv\") pod \"ai-gateway-controller-dcd84bcbc-k6pdq\" (UID: \"17f4db95-d03a-4200-8b01-d7760251e7fa\") " pod="opendatahub/ai-gateway-controller-dcd84bcbc-k6pdq"
Nov 02 12:06:25.743903 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:25.743876    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-dcd84bcbc-k6pdq"
Nov 02 12:06:26.112893 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:26.112865    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-dcd84bcbc-k6pdq"]
Nov 02 12:06:26.114469 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:06:26.114442    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f4db95_d03a_4200_8b01_d7760251e7fa.slice/crio-00dc2a079f7fcc32cfe7ec67a64ed3ffc303955f751851db1c5fb59cb12d5408 WatchSource:0}: Error finding container 00dc2a079f7fcc32cfe7ec67a64ed3ffc303955f751851db1c5fb59cb12d5408: Status 404 returned error can't find the container with id 00dc2a079f7fcc32cfe7ec67a64ed3ffc303955f751851db1c5fb59cb12d5408
Nov 02 12:06:26.124634 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:26.124606    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-dcd84bcbc-k6pdq" event={"ID":"17f4db95-d03a-4200-8b01-d7760251e7fa","Type":"ContainerStarted","Data":"00dc2a079f7fcc32cfe7ec67a64ed3ffc303955f751851db1c5fb59cb12d5408"}
Nov 02 12:06:27.130260 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:27.130224    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-dcd84bcbc-k6pdq" event={"ID":"17f4db95-d03a-4200-8b01-d7760251e7fa","Type":"ContainerStarted","Data":"1416385ace2eda8e78260dd54c05ed9029de02bf88d62dab085a057c6b0604f8"}
Nov 02 12:06:27.130608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:27.130337    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-dcd84bcbc-k6pdq"
Nov 02 12:06:27.251192 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:27.251123    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-dcd84bcbc-k6pdq" podStartSLOduration=1.476079137 podStartE2EDuration="2.251106251s" totalImagesPullingTime="775.027114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:06:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:06:27.24878889 +0000 UTC m=+615.812497300" watchObservedRunningTime="2026-11-02 12:06:27.251106251 +0000 UTC m=+615.814814661"
Nov 02 12:06:28.291185 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:06:28.291148    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:06:33.113877 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:33.113850    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"
Nov 02 12:06:33.114344 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:33.113914    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"
Nov 02 12:06:38.136952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:38.136918    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-dcd84bcbc-k6pdq"
Nov 02 12:06:38.309803 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:38.309773    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2xzcb"]
Nov 02 12:06:38.310035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:38.309990    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb" podUID="334bc218-c5a6-41ac-bf0b-2818e2033c13" containerName="manager" containerID="cri-o://1aaf47510a13fd698670a80ae1139fd9aeb48ee30237cbc35a22e4e0a21e9a16" gracePeriod=30
Nov 02 12:06:38.552621 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:38.552595    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb"
Nov 02 12:06:38.649910 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:38.649873    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/334bc218-c5a6-41ac-bf0b-2818e2033c13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/334bc218-c5a6-41ac-bf0b-2818e2033c13-metrics-certs\") pod \"334bc218-c5a6-41ac-bf0b-2818e2033c13\" (UID: \"334bc218-c5a6-41ac-bf0b-2818e2033c13\") "
Nov 02 12:06:38.650116 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:38.649958    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334bc218-c5a6-41ac-bf0b-2818e2033c13-kube-api-access-4mp4j\" (UniqueName: \"kubernetes.io/projected/334bc218-c5a6-41ac-bf0b-2818e2033c13-kube-api-access-4mp4j\") pod \"334bc218-c5a6-41ac-bf0b-2818e2033c13\" (UID: \"334bc218-c5a6-41ac-bf0b-2818e2033c13\") "
Nov 02 12:06:38.651965 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:38.651941    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334bc218-c5a6-41ac-bf0b-2818e2033c13-metrics-certs" pod "334bc218-c5a6-41ac-bf0b-2818e2033c13" (UID: "334bc218-c5a6-41ac-bf0b-2818e2033c13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:06:38.652151 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:38.652129    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334bc218-c5a6-41ac-bf0b-2818e2033c13-kube-api-access-4mp4j" pod "334bc218-c5a6-41ac-bf0b-2818e2033c13" (UID: "334bc218-c5a6-41ac-bf0b-2818e2033c13"). InnerVolumeSpecName "kube-api-access-4mp4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:06:38.751154 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:38.751125    4464 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/334bc218-c5a6-41ac-bf0b-2818e2033c13-metrics-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:06:38.751154 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:38.751150    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mp4j\" (UniqueName: \"kubernetes.io/projected/334bc218-c5a6-41ac-bf0b-2818e2033c13-kube-api-access-4mp4j\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:06:39.185150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:39.185063    4464 generic.go:356] "Generic (PLEG): container finished" podID="334bc218-c5a6-41ac-bf0b-2818e2033c13" containerID="1aaf47510a13fd698670a80ae1139fd9aeb48ee30237cbc35a22e4e0a21e9a16" exitCode=0
Nov 02 12:06:39.185150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:39.185115    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb"
Nov 02 12:06:39.185594 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:39.185147    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb" event={"ID":"334bc218-c5a6-41ac-bf0b-2818e2033c13","Type":"ContainerDied","Data":"1aaf47510a13fd698670a80ae1139fd9aeb48ee30237cbc35a22e4e0a21e9a16"}
Nov 02 12:06:39.185594 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:39.185202    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-2xzcb" event={"ID":"334bc218-c5a6-41ac-bf0b-2818e2033c13","Type":"ContainerDied","Data":"04fc1112cb1afc29044427a97723de7a8291edd9024f012b7b5ba21e5ce223d3"}
Nov 02 12:06:39.185594 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:39.185225    4464 scope.go:122] "RemoveContainer" containerID="1aaf47510a13fd698670a80ae1139fd9aeb48ee30237cbc35a22e4e0a21e9a16"
Nov 02 12:06:39.194073 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:39.194057    4464 scope.go:122] "RemoveContainer" containerID="1aaf47510a13fd698670a80ae1139fd9aeb48ee30237cbc35a22e4e0a21e9a16"
Nov 02 12:06:39.194341 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:06:39.194317    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aaf47510a13fd698670a80ae1139fd9aeb48ee30237cbc35a22e4e0a21e9a16\": container with ID starting with 1aaf47510a13fd698670a80ae1139fd9aeb48ee30237cbc35a22e4e0a21e9a16 not found: ID does not exist" containerID="1aaf47510a13fd698670a80ae1139fd9aeb48ee30237cbc35a22e4e0a21e9a16"
Nov 02 12:06:39.194418 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:39.194353    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aaf47510a13fd698670a80ae1139fd9aeb48ee30237cbc35a22e4e0a21e9a16"} err="failed to get container status \"1aaf47510a13fd698670a80ae1139fd9aeb48ee30237cbc35a22e4e0a21e9a16\": rpc error: code = NotFound desc = could not find container \"1aaf47510a13fd698670a80ae1139fd9aeb48ee30237cbc35a22e4e0a21e9a16\": container with ID starting with 1aaf47510a13fd698670a80ae1139fd9aeb48ee30237cbc35a22e4e0a21e9a16 not found: ID does not exist"
Nov 02 12:06:39.352611 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:39.352577    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2xzcb"]
Nov 02 12:06:39.403520 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:39.403491    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2xzcb"]
Nov 02 12:06:39.580496 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:39.580466    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334bc218-c5a6-41ac-bf0b-2818e2033c13" path="/var/lib/kubelet/pods/334bc218-c5a6-41ac-bf0b-2818e2033c13/volumes"
Nov 02 12:06:49.203386 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.203353    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"]
Nov 02 12:06:49.203776 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.203763    4464 cpu_manager.go:414] "containerMap: removing container" podUID="334bc218-c5a6-41ac-bf0b-2818e2033c13" containerName="manager"
Nov 02 12:06:49.203818 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.203776    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334bc218-c5a6-41ac-bf0b-2818e2033c13" containerName="manager"
Nov 02 12:06:49.203856 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.203851    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="334bc218-c5a6-41ac-bf0b-2818e2033c13" containerName="manager"
Nov 02 12:06:49.208948 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.208928    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.265033 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.265007    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 12:06:49.265180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.265142    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 12:06:49.265180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.265149    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-p8m6b\""
Nov 02 12:06:49.265180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.265177    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 02 12:06:49.317655 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.317615    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"]
Nov 02 12:06:49.343264 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.343234    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.343402 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.343279    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.343444 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.343404    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.343444 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.343431    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mtmh\" (UniqueName: \"kubernetes.io/projected/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-kube-api-access-6mtmh\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.343548 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.343506    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.343599 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.343552    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.444726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.444692    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.444726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.444731    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.444945 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.444769    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.444945 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.444791    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.444945 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.444849    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.444945 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.444875    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mtmh\" (UniqueName: \"kubernetes.io/projected/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-kube-api-access-6mtmh\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.445288 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.445262    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.445404 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.445342    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.445404 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.445362    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.447073 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.447046    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.447276 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.447257    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.520667 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.520642    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mtmh\" (UniqueName: \"kubernetes.io/projected/6a1bdd91-9d36-4c44-a3ac-694afddf8ba0-kube-api-access-6mtmh\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7\" (UID: \"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:49.820042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:49.819942    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:50.191001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:50.190961    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"]
Nov 02 12:06:50.192941 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:06:50.192912    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1bdd91_9d36_4c44_a3ac_694afddf8ba0.slice/crio-dd3299fb5f85616f660030a106c69600bbb7fb4690cff752b8299501f29807a7 WatchSource:0}: Error finding container dd3299fb5f85616f660030a106c69600bbb7fb4690cff752b8299501f29807a7: Status 404 returned error can't find the container with id dd3299fb5f85616f660030a106c69600bbb7fb4690cff752b8299501f29807a7
Nov 02 12:06:50.239924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:50.239883    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7" event={"ID":"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0","Type":"ContainerStarted","Data":"dd3299fb5f85616f660030a106c69600bbb7fb4690cff752b8299501f29807a7"}
Nov 02 12:06:52.751845 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:52.751814    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:06:53.257535 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:53.257501    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7" event={"ID":"6a1bdd91-9d36-4c44-a3ac-694afddf8ba0","Type":"ContainerStarted","Data":"163097f05ebe4550775ce062df9bdcb19facace79b9d54e9985f559717b26484"}
Nov 02 12:06:53.378445 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:53.378391    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7" podStartSLOduration=2.307481479 podStartE2EDuration="4.378376761s" totalImagesPullingTime="2.070895282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:06:53.377382247 +0000 UTC m=+641.941090661" watchObservedRunningTime="2026-11-02 12:06:53.378376761 +0000 UTC m=+641.942085169"
Nov 02 12:06:53.820621 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:53.820587    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:55.252663 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.252630    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"]
Nov 02 12:06:55.255059 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.255043    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.364068 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.364041    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 02 12:06:55.365666 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.365643    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"]
Nov 02 12:06:55.402837 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.402806    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53517c94-5480-4ee7-abcf-cdc03f8af0a1-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.402837 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.402836    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53517c94-5480-4ee7-abcf-cdc03f8af0a1-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.403082 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.402936    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53517c94-5480-4ee7-abcf-cdc03f8af0a1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.403082 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.402965    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53517c94-5480-4ee7-abcf-cdc03f8af0a1-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.403210 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.403084    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzn9t\" (UniqueName: \"kubernetes.io/projected/53517c94-5480-4ee7-abcf-cdc03f8af0a1-kube-api-access-tzn9t\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.403210 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.403104    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53517c94-5480-4ee7-abcf-cdc03f8af0a1-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.504216 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.504139    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53517c94-5480-4ee7-abcf-cdc03f8af0a1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.504216 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.504178    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53517c94-5480-4ee7-abcf-cdc03f8af0a1-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.504431 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.504221    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzn9t\" (UniqueName: \"kubernetes.io/projected/53517c94-5480-4ee7-abcf-cdc03f8af0a1-kube-api-access-tzn9t\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.504431 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.504240    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53517c94-5480-4ee7-abcf-cdc03f8af0a1-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.504431 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.504280    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53517c94-5480-4ee7-abcf-cdc03f8af0a1-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.504431 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.504396    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53517c94-5480-4ee7-abcf-cdc03f8af0a1-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.504618 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.504582    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53517c94-5480-4ee7-abcf-cdc03f8af0a1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.504680 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.504645    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53517c94-5480-4ee7-abcf-cdc03f8af0a1-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.504735 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.504690    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53517c94-5480-4ee7-abcf-cdc03f8af0a1-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.506489 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.506465    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53517c94-5480-4ee7-abcf-cdc03f8af0a1-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.506783 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.506762    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53517c94-5480-4ee7-abcf-cdc03f8af0a1-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.566282 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.566247    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzn9t\" (UniqueName: \"kubernetes.io/projected/53517c94-5480-4ee7-abcf-cdc03f8af0a1-kube-api-access-tzn9t\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8j57g\" (UID: \"53517c94-5480-4ee7-abcf-cdc03f8af0a1\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:55.866605 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:55.866519    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:56.237873 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:56.237848    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"]
Nov 02 12:06:56.239570 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:06:56.239541    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53517c94_5480_4ee7_abcf_cdc03f8af0a1.slice/crio-c3992ded9f94f7e0bda480e8e09d379a24c083653e6e86cbcbdc25ea84375261 WatchSource:0}: Error finding container c3992ded9f94f7e0bda480e8e09d379a24c083653e6e86cbcbdc25ea84375261: Status 404 returned error can't find the container with id c3992ded9f94f7e0bda480e8e09d379a24c083653e6e86cbcbdc25ea84375261
Nov 02 12:06:56.278731 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:56.278697    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g" event={"ID":"53517c94-5480-4ee7-abcf-cdc03f8af0a1","Type":"ContainerStarted","Data":"c3992ded9f94f7e0bda480e8e09d379a24c083653e6e86cbcbdc25ea84375261"}
Nov 02 12:06:57.285377 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:57.285343    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g" event={"ID":"53517c94-5480-4ee7-abcf-cdc03f8af0a1","Type":"ContainerStarted","Data":"d3b661b64e838f737339b9db9f30f01819afdcdf1e56017b5464c610a061847c"}
Nov 02 12:06:57.404626 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:57.404574    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g" podStartSLOduration=1.8229107230000001 podStartE2EDuration="2.404559564s" totalImagesPullingTime="581.648841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:06:57.403940855 +0000 UTC m=+645.967649265" watchObservedRunningTime="2026-11-02 12:06:57.404559564 +0000 UTC m=+645.968267973"
Nov 02 12:06:57.867131 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:57.867096    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:06:58.292178 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:06:58.292148    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:06:59.820397 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:59.820365    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:06:59.832993 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:06:59.832960    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:07:00.198240 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.198154    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"]
Nov 02 12:07:00.201821 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.201798    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.255124 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.255092    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 02 12:07:00.260927 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.260887    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"]
Nov 02 12:07:00.309857 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.309810    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7"
Nov 02 12:07:00.349143 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.349110    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55c46ddc-9659-48d9-92c6-d8363f573813-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.349143 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.349149    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55c46ddc-9659-48d9-92c6-d8363f573813-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.349327 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.349243    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wd2k\" (UniqueName: \"kubernetes.io/projected/55c46ddc-9659-48d9-92c6-d8363f573813-kube-api-access-8wd2k\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.349327 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.349274    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55c46ddc-9659-48d9-92c6-d8363f573813-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.349394 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.349367    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55c46ddc-9659-48d9-92c6-d8363f573813-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.349424 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.349412    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55c46ddc-9659-48d9-92c6-d8363f573813-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.450729 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.450650    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55c46ddc-9659-48d9-92c6-d8363f573813-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.450729 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.450684    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55c46ddc-9659-48d9-92c6-d8363f573813-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.450729 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.450712    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55c46ddc-9659-48d9-92c6-d8363f573813-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.451013 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.450745    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55c46ddc-9659-48d9-92c6-d8363f573813-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.451013 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.450818    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wd2k\" (UniqueName: \"kubernetes.io/projected/55c46ddc-9659-48d9-92c6-d8363f573813-kube-api-access-8wd2k\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.451013 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.450841    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55c46ddc-9659-48d9-92c6-d8363f573813-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.451178 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.451129    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55c46ddc-9659-48d9-92c6-d8363f573813-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.451234 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.451195    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55c46ddc-9659-48d9-92c6-d8363f573813-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.451286 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.451266    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55c46ddc-9659-48d9-92c6-d8363f573813-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.453132 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.453105    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55c46ddc-9659-48d9-92c6-d8363f573813-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.453481 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.453465    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55c46ddc-9659-48d9-92c6-d8363f573813-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.512706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.512675    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wd2k\" (UniqueName: \"kubernetes.io/projected/55c46ddc-9659-48d9-92c6-d8363f573813-kube-api-access-8wd2k\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-r8qcw\" (UID: \"55c46ddc-9659-48d9-92c6-d8363f573813\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.514591 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.514566    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:00.886644 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:00.886620    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"]
Nov 02 12:07:00.888647 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:07:00.888621    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c46ddc_9659_48d9_92c6_d8363f573813.slice/crio-9b3dc8eab04d74d5747372e17ddda664369852cce26fca09bab9ad0f045b9195 WatchSource:0}: Error finding container 9b3dc8eab04d74d5747372e17ddda664369852cce26fca09bab9ad0f045b9195: Status 404 returned error can't find the container with id 9b3dc8eab04d74d5747372e17ddda664369852cce26fca09bab9ad0f045b9195
Nov 02 12:07:01.303867 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:01.303827    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw" event={"ID":"55c46ddc-9659-48d9-92c6-d8363f573813","Type":"ContainerStarted","Data":"9b3dc8eab04d74d5747372e17ddda664369852cce26fca09bab9ad0f045b9195"}
Nov 02 12:07:02.309668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:02.309629    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw" event={"ID":"55c46ddc-9659-48d9-92c6-d8363f573813","Type":"ContainerStarted","Data":"3d34f7edb7c9c52838e37bdd5e547035e10d7aa19920acd8a920331e2b1a284c"}
Nov 02 12:07:02.430007 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:02.429940    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw" podStartSLOduration=1.840469241 podStartE2EDuration="2.429928377s" totalImagesPullingTime="589.459136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:07:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:07:02.428410777 +0000 UTC m=+650.992119189" watchObservedRunningTime="2026-11-02 12:07:02.429928377 +0000 UTC m=+650.993636786"
Nov 02 12:07:02.515154 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:02.515120    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:03.773962 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:03.773925    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:07:05.867647 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:05.867609    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:07:05.879864 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:05.879838    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:07:06.340818 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:06.340786    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8j57g"
Nov 02 12:07:10.515580 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:10.515548    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:10.528041 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:10.528015    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:11.364719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:11.364689    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-r8qcw"
Nov 02 12:07:12.310707 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.310672    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-8bkgg"]
Nov 02 12:07:12.316630 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.316611    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.358889 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.358854    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsdpj\" (UniqueName: \"kubernetes.io/projected/c1851d27-686d-4b50-bbf6-db2e372dd09c-kube-api-access-nsdpj\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.359069 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.358901    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1851d27-686d-4b50-bbf6-db2e372dd09c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.359122 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.359084    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1851d27-686d-4b50-bbf6-db2e372dd09c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.359171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.359116    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1851d27-686d-4b50-bbf6-db2e372dd09c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.359171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.359158    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1851d27-686d-4b50-bbf6-db2e372dd09c-home\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.359266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.359244    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1851d27-686d-4b50-bbf6-db2e372dd09c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.422134 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.422102    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 02 12:07:12.423813 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.423790    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-8bkgg"]
Nov 02 12:07:12.459991 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.459945    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsdpj\" (UniqueName: \"kubernetes.io/projected/c1851d27-686d-4b50-bbf6-db2e372dd09c-kube-api-access-nsdpj\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.460137 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.459999    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1851d27-686d-4b50-bbf6-db2e372dd09c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.460137 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.460095    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1851d27-686d-4b50-bbf6-db2e372dd09c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.460137 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.460123    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1851d27-686d-4b50-bbf6-db2e372dd09c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.460327 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.460161    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1851d27-686d-4b50-bbf6-db2e372dd09c-home\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.460327 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.460187    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1851d27-686d-4b50-bbf6-db2e372dd09c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.460586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.460554    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1851d27-686d-4b50-bbf6-db2e372dd09c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.460586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.460572    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1851d27-686d-4b50-bbf6-db2e372dd09c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.460756 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.460715    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1851d27-686d-4b50-bbf6-db2e372dd09c-home\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.462554 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.462519    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1851d27-686d-4b50-bbf6-db2e372dd09c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.462839 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.462810    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1851d27-686d-4b50-bbf6-db2e372dd09c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.525037 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.525007    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsdpj\" (UniqueName: \"kubernetes.io/projected/c1851d27-686d-4b50-bbf6-db2e372dd09c-kube-api-access-nsdpj\") pod \"echo-simulator-kserve-64b6f8f96c-8bkgg\" (UID: \"c1851d27-686d-4b50-bbf6-db2e372dd09c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:12.613602 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.613518    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:07:12.629139 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:12.629108    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:13.001848 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.001823    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-8bkgg"]
Nov 02 12:07:13.003594 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:07:13.003564    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1851d27_686d_4b50_bbf6_db2e372dd09c.slice/crio-5a431620a3102f516045de96fd182003faf71c4d936ccbee8b7abf49efb3db03 WatchSource:0}: Error finding container 5a431620a3102f516045de96fd182003faf71c4d936ccbee8b7abf49efb3db03: Status 404 returned error can't find the container with id 5a431620a3102f516045de96fd182003faf71c4d936ccbee8b7abf49efb3db03
Nov 02 12:07:13.365161 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.365082    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg" event={"ID":"c1851d27-686d-4b50-bbf6-db2e372dd09c","Type":"ContainerStarted","Data":"5a431620a3102f516045de96fd182003faf71c4d936ccbee8b7abf49efb3db03"}
Nov 02 12:07:13.438842 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.438811    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"]
Nov 02 12:07:13.444335 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.444309    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.468053 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.468010    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvfc9\" (UniqueName: \"kubernetes.io/projected/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-kube-api-access-lvfc9\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.468223 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.468098    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.468223 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.468184    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.468223 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.468219    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.468377 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.468242    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.468425 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.468387    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.468472 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.468446    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.468472 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.468465    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.468550 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.468498    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.548465 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.548426    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c5ldd\""
Nov 02 12:07:13.551109 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.551083    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"]
Nov 02 12:07:13.569738 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.569711    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.569901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.569742    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.569901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.569762    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.569901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.569797    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.569901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.569820    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvfc9\" (UniqueName: \"kubernetes.io/projected/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-kube-api-access-lvfc9\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.569901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.569839    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.569901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.569901    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.570250 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.569935    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.570250 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.569959    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.570250 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.570196    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.570408 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.570315    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.570464 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.570403    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.570511 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.570476    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.570634 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.570610    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.572275 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.572246    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.572555 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.572537    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.636479 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.636377    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.636613 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.636565    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvfc9\" (UniqueName: \"kubernetes.io/projected/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-kube-api-access-lvfc9\") pod \"maas-default-gateway-openshift-default-6dcf757899-ddhbn\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:13.760748 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:13.760693    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:14.188433 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:14.188406    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"]
Nov 02 12:07:14.191199 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:07:14.191149    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4888bcb6_bcf0_4c46_9e89_bc4f1feb6df4.slice/crio-bc04544cacb0a643044c4d7926c9b3abff7dfaa0f7119c9d148716d0cad388e8 WatchSource:0}: Error finding container bc04544cacb0a643044c4d7926c9b3abff7dfaa0f7119c9d148716d0cad388e8: Status 404 returned error can't find the container with id bc04544cacb0a643044c4d7926c9b3abff7dfaa0f7119c9d148716d0cad388e8
Nov 02 12:07:14.371556 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:14.371527    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn" event={"ID":"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4","Type":"ContainerStarted","Data":"bc04544cacb0a643044c4d7926c9b3abff7dfaa0f7119c9d148716d0cad388e8"}
Nov 02 12:07:16.384005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:16.383918    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:07:16.384005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:16.383957    4464 generic.go:356] "Generic (PLEG): container finished" podID="c1851d27-686d-4b50-bbf6-db2e372dd09c" containerID="bd574b767ff8729a6688ae56586c036fa4503c73ca55e0a448d1601d9766d157" exitCode=137
Nov 02 12:07:16.384477 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:16.384050    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg" event={"ID":"c1851d27-686d-4b50-bbf6-db2e372dd09c","Type":"ContainerDied","Data":"bd574b767ff8729a6688ae56586c036fa4503c73ca55e0a448d1601d9766d157"}
Nov 02 12:07:16.384477 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:16.384362    4464 scope.go:122] "RemoveContainer" containerID="bd574b767ff8729a6688ae56586c036fa4503c73ca55e0a448d1601d9766d157"
Nov 02 12:07:16.630243 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:16.630201    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:17.392560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:17.392490    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:07:17.393005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:17.392567    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg" event={"ID":"c1851d27-686d-4b50-bbf6-db2e372dd09c","Type":"ContainerStarted","Data":"f8926c6e0f568c11e41b9228430e514aaba3657c2c862c29bd879478087a47d9"}
Nov 02 12:07:17.597393 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:17.597162    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg" podStartSLOduration=2.408183201 podStartE2EDuration="5.597147576s" totalImagesPullingTime="3.188964375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:07:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:07:17.595350211 +0000 UTC m=+666.159058621" watchObservedRunningTime="2026-11-02 12:07:17.597147576 +0000 UTC m=+666.160855984"
Nov 02 12:07:17.629346 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:17.629315    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:17.811806 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:17.811774    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:07:17.811907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:17.811846    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:07:17.811907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:17.811873    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:07:18.398802 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:18.398753    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn" event={"ID":"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4","Type":"ContainerStarted","Data":"6bf1c07240323add86bb44f0127a2f664f57b5a2afdc5eacbe75cbf2154e5aa3"}
Nov 02 12:07:18.522305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:18.522249    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn" podStartSLOduration=1.9053111889999998 podStartE2EDuration="5.522235868s" totalImagesPullingTime="3.616924679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:07:18.520708246 +0000 UTC m=+667.084416667" watchObservedRunningTime="2026-11-02 12:07:18.522235868 +0000 UTC m=+667.085944350"
Nov 02 12:07:18.777627 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:18.769348    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:18.780157 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:18.780121    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ddhbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 12:07:18.780384 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:18.780357    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn" podUID="4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 12:07:19.768010 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:19.767412    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ddhbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 12:07:19.768010 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:19.767491    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn" podUID="4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 12:07:20.763415 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:20.763376    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ddhbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 12:07:20.763736 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:20.763696    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn" podUID="4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 12:07:21.764002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:21.763360    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ddhbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 12:07:21.764002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:21.763423    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn" podUID="4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 12:07:22.630295 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:22.630260    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:22.642749 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:22.642725    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:22.784386 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:22.784355    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ddhbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 12:07:22.784800 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:22.784409    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn" podUID="4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 12:07:23.432068 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:23.432041    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-8bkgg"
Nov 02 12:07:23.761591 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:23.761555    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ddhbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 12:07:23.761760 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:23.761620    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn" podUID="4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 12:07:24.761528 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:24.761450    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ddhbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 02 12:07:24.761892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:24.761528    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn" podUID="4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 02 12:07:26.687236 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:26.687204    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:07:26.761270 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:26.761216    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ddhbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:07:26.761488 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:26.761301    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn" podUID="4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:07:27.803161 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:27.803115    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ddhbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:07:27.803683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:27.803212    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn" podUID="4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:07:28.293317 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:07:28.293273    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:07:28.372262 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.372234    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:28.372554 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.372531    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:28.373307 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.373289    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:07:28.579819 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.579740    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"]
Nov 02 12:07:28.583154 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.583137    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.681794 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.681761    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"]
Nov 02 12:07:28.738582 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.738549    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.738582 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.738586    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.738783 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.738704    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnjtd\" (UniqueName: \"kubernetes.io/projected/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-kube-api-access-nnjtd\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.738783 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.738743    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.738897 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.738868    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.738951 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.738921    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.739024 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.739006    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.739069 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.739038    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.739172 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.739152    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.840673 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.840587    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnjtd\" (UniqueName: \"kubernetes.io/projected/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-kube-api-access-nnjtd\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.840673 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.840637    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.841228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.840677    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.841228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.840696    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.841228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.840737    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.841228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.840887    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.841228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.840991    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.841228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.841037    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.841228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.841071    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.841228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.841195    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.841228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.841227    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.841660 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.841440    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.841660 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.841491    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.842076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.842017    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.843773 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.843750    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.843867 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.843829    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.903131 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.903093    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:28.903267 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:28.903200    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnjtd\" (UniqueName: \"kubernetes.io/projected/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-kube-api-access-nnjtd\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxp92\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:29.196135 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:29.196050    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:29.622292 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:29.622253    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"]
Nov 02 12:07:29.625956 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:07:29.625924    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1754568_1a2e_4c3a_9a37_2a80ef54ab00.slice/crio-8aac853377bf62b43594265c9b7a902941747e07e765ada05ba858a586bb5a53 WatchSource:0}: Error finding container 8aac853377bf62b43594265c9b7a902941747e07e765ada05ba858a586bb5a53: Status 404 returned error can't find the container with id 8aac853377bf62b43594265c9b7a902941747e07e765ada05ba858a586bb5a53
Nov 02 12:07:29.629714 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:29.629683    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:07:29.629797 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:29.629752    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:07:29.629797 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:29.629782    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:07:30.455895 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:30.455857    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92" event={"ID":"a1754568-1a2e-4c3a-9a37-2a80ef54ab00","Type":"ContainerStarted","Data":"bf42302a90351fedea4b1383ce3b220bdf49ac6643ab4181a04f90d51c495946"}
Nov 02 12:07:30.456323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:30.455916    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92" event={"ID":"a1754568-1a2e-4c3a-9a37-2a80ef54ab00","Type":"ContainerStarted","Data":"8aac853377bf62b43594265c9b7a902941747e07e765ada05ba858a586bb5a53"}
Nov 02 12:07:31.196791 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:31.196755    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:31.198175 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:31.198148    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mxp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:07:31.198315 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:31.198195    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92" podUID="a1754568-1a2e-4c3a-9a37-2a80ef54ab00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 12:07:32.197004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:32.196944    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mxp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:07:32.197420 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:32.197028    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92" podUID="a1754568-1a2e-4c3a-9a37-2a80ef54ab00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 12:07:33.196940 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:33.196900    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mxp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:07:33.197152 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:33.196958    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92" podUID="a1754568-1a2e-4c3a-9a37-2a80ef54ab00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 12:07:34.197239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:34.197201    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mxp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:07:34.197628 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:34.197256    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92" podUID="a1754568-1a2e-4c3a-9a37-2a80ef54ab00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 12:07:35.197292 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:35.197254    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mxp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:07:35.197678 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:35.197318    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92" podUID="a1754568-1a2e-4c3a-9a37-2a80ef54ab00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 12:07:36.196780 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:36.196742    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mxp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:07:36.196943 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:36.196801    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92" podUID="a1754568-1a2e-4c3a-9a37-2a80ef54ab00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 12:07:37.196894 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:37.196856    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mxp92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:07:37.197293 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:37.196939    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92" podUID="a1754568-1a2e-4c3a-9a37-2a80ef54ab00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 12:07:38.839572 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:38.839539    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:38.840035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:38.839947    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:38.840780 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:38.840761    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:07:39.012655 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:07:39.012603    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92" podStartSLOduration=11.012590201 podStartE2EDuration="11.012590201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:07:30.584206329 +0000 UTC m=+679.147914738" watchObservedRunningTime="2026-11-02 12:07:39.012590201 +0000 UTC m=+687.576298609"
Nov 02 12:07:58.291456 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:07:58.291415    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:08:26.098348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.098312    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"]
Nov 02 12:08:26.101219 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.101202    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.161229 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.161197    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4b9210-openshift-default-dockercfg-fqcg6\""
Nov 02 12:08:26.195083 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.195052    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.195522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.195102    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.195522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.195189    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.195522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.195240    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgkk5\" (UniqueName: \"kubernetes.io/projected/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-kube-api-access-vgkk5\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.195522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.195270    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.195522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.195291    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.195522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.195339    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.195522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.195389    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.195522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.195498    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.214988 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.214942    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"]
Nov 02 12:08:26.297070 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.297036    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.297070 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.297078    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.297323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.297121    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgkk5\" (UniqueName: \"kubernetes.io/projected/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-kube-api-access-vgkk5\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.297323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.297161    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.297323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.297191    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.297323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.297216    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.297323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.297240    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.297323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.297291    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.297629 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.297386    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.297629 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.297586    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.297732 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.297651    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.297924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.297900    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.297924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.297913    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.298063 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.298020    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.299445 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.299421    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.299802 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.299782    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.359761 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.359690    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgkk5\" (UniqueName: \"kubernetes.io/projected/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-kube-api-access-vgkk5\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.360223 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.360200    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.412764 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.412731    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:26.848631 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.848590    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"]
Nov 02 12:08:26.851012 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:08:26.850987    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d29abcb_5367_49e7_b4b0_4c27e87fe2a7.slice/crio-7b24aec605dfcb01a3bb052fc89a77a858a6f816772da962268914346e087794 WatchSource:0}: Error finding container 7b24aec605dfcb01a3bb052fc89a77a858a6f816772da962268914346e087794: Status 404 returned error can't find the container with id 7b24aec605dfcb01a3bb052fc89a77a858a6f816772da962268914346e087794
Nov 02 12:08:26.853069 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.853041    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:08:26.853127 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.853102    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:08:26.853163 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:26.853132    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:08:27.383447 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.383414    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"]
Nov 02 12:08:27.386022 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.385992    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.409293 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.409270    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739b7556-be7f-4208-b953-4f82f7511f3e-istio-podinfo\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.409426 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.409304    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-istio-data\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.409426 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.409325    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739b7556-be7f-4208-b953-4f82f7511f3e-istiod-ca-cert\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.409426 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.409347    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-workload-certs\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.409553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.409475    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl4pm\" (UniqueName: \"kubernetes.io/projected/739b7556-be7f-4208-b953-4f82f7511f3e-kube-api-access-fl4pm\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.409553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.409512    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-istio-envoy\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.409626 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.409569    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-credential-socket\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.409626 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.409592    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-workload-socket\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.409770 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.409752    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/739b7556-be7f-4208-b953-4f82f7511f3e-istio-token\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.462713 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.462683    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b6f3d8-openshift-default-dockercfg-lghc5\""
Nov 02 12:08:27.479557 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.479527    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"]
Nov 02 12:08:27.510749 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.510716    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl4pm\" (UniqueName: \"kubernetes.io/projected/739b7556-be7f-4208-b953-4f82f7511f3e-kube-api-access-fl4pm\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.510903 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.510755    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-istio-envoy\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.510903 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.510780    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-credential-socket\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.510903 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.510798    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-workload-socket\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.511093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.510922    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/739b7556-be7f-4208-b953-4f82f7511f3e-istio-token\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.511093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.510958    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739b7556-be7f-4208-b953-4f82f7511f3e-istio-podinfo\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.511093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.511023    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-istio-data\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.511093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.511046    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739b7556-be7f-4208-b953-4f82f7511f3e-istiod-ca-cert\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.511093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.511075    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-workload-certs\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.511350 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.511303    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-workload-socket\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.511513 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.511483    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-workload-certs\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.511648 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.511495    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-credential-socket\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.511648 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.511527    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-istio-data\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.511969 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.511949    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739b7556-be7f-4208-b953-4f82f7511f3e-istiod-ca-cert\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.513200 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.513174    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-istio-envoy\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.513445 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.513421    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739b7556-be7f-4208-b953-4f82f7511f3e-istio-podinfo\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.575099 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.575043    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl4pm\" (UniqueName: \"kubernetes.io/projected/739b7556-be7f-4208-b953-4f82f7511f3e-kube-api-access-fl4pm\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.580703 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.580677    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/739b7556-be7f-4208-b953-4f82f7511f3e-istio-token\") pod \"e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") " pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.700851 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.697879    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:27.743018 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.742957    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" event={"ID":"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7","Type":"ContainerStarted","Data":"886d0c3fe1f711852cb41aa5416170ccc082175ccb5d914f176925e7349f7fcd"}
Nov 02 12:08:27.743185 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.743048    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" event={"ID":"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7","Type":"ContainerStarted","Data":"7b24aec605dfcb01a3bb052fc89a77a858a6f816772da962268914346e087794"}
Nov 02 12:08:27.868260 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:27.868197    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" podStartSLOduration=1.868178981 podStartE2EDuration="1.868178981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:08:27.866729571 +0000 UTC m=+736.430438004" watchObservedRunningTime="2026-11-02 12:08:27.868178981 +0000 UTC m=+736.431887394"
Nov 02 12:08:28.130942 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:28.130915    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"]
Nov 02 12:08:28.133109 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:08:28.133082    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739b7556_be7f_4208_b953_4f82f7511f3e.slice/crio-fc0a5f42059e4330acc47cda9d8e21647fb55742a5e840d14a75954970f79d8a WatchSource:0}: Error finding container fc0a5f42059e4330acc47cda9d8e21647fb55742a5e840d14a75954970f79d8a: Status 404 returned error can't find the container with id fc0a5f42059e4330acc47cda9d8e21647fb55742a5e840d14a75954970f79d8a
Nov 02 12:08:28.135392 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:28.135356    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:08:28.135505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:28.135438    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:08:28.135505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:28.135474    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:08:28.292575 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:08:28.292530    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:08:28.413165 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:28.413080    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:28.417705 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:28.417673    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:28.749284 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:28.749248    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9" event={"ID":"739b7556-be7f-4208-b953-4f82f7511f3e","Type":"ContainerStarted","Data":"a5eddea6d73d2504c8942f3c73e7ac577b68931e325a29cd44dedc1bb580f712"}
Nov 02 12:08:28.749448 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:28.749303    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9" event={"ID":"739b7556-be7f-4208-b953-4f82f7511f3e","Type":"ContainerStarted","Data":"fc0a5f42059e4330acc47cda9d8e21647fb55742a5e840d14a75954970f79d8a"}
Nov 02 12:08:28.749578 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:28.749561    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:28.750294 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:28.750274    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:28.872993 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:28.872938    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9" podStartSLOduration=1.872926796 podStartE2EDuration="1.872926796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:08:28.871201348 +0000 UTC m=+737.434909773" watchObservedRunningTime="2026-11-02 12:08:28.872926796 +0000 UTC m=+737.436635204"
Nov 02 12:08:29.699177 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:29.699139    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:29.704077 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:29.704056    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:29.754357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:29.754330    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:29.754918 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:29.754903    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:08:34.688346 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.688317    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"]
Nov 02 12:08:34.690755 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.690738    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"
Nov 02 12:08:34.787892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.787858    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964e4b45-70c3-4ab2-8960-cf5f56c36a04-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l\" (UID: \"964e4b45-70c3-4ab2-8960-cf5f56c36a04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"
Nov 02 12:08:34.788131 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.788085    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zff8\" (UniqueName: \"kubernetes.io/projected/964e4b45-70c3-4ab2-8960-cf5f56c36a04-kube-api-access-8zff8\") pod \"payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l\" (UID: \"964e4b45-70c3-4ab2-8960-cf5f56c36a04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"
Nov 02 12:08:34.794803 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.794772    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-b1703c-dockercfg-wsg2w\""
Nov 02 12:08:34.794943 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.794838    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b1703c\""
Nov 02 12:08:34.798551 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.798431    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"]
Nov 02 12:08:34.827988 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.827942    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"]
Nov 02 12:08:34.830593 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.830573    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"
Nov 02 12:08:34.889478 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.889451    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zff8\" (UniqueName: \"kubernetes.io/projected/964e4b45-70c3-4ab2-8960-cf5f56c36a04-kube-api-access-8zff8\") pod \"payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l\" (UID: \"964e4b45-70c3-4ab2-8960-cf5f56c36a04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"
Nov 02 12:08:34.889612 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.889505    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e85c3283-ce74-4e7f-b191-79e497d42a77-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h\" (UID: \"e85c3283-ce74-4e7f-b191-79e497d42a77\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"
Nov 02 12:08:34.889612 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.889575    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964e4b45-70c3-4ab2-8960-cf5f56c36a04-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l\" (UID: \"964e4b45-70c3-4ab2-8960-cf5f56c36a04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"
Nov 02 12:08:34.889734 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.889611    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pn54\" (UniqueName: \"kubernetes.io/projected/e85c3283-ce74-4e7f-b191-79e497d42a77-kube-api-access-7pn54\") pod \"payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h\" (UID: \"e85c3283-ce74-4e7f-b191-79e497d42a77\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"
Nov 02 12:08:34.890322 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.890301    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964e4b45-70c3-4ab2-8960-cf5f56c36a04-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l\" (UID: \"964e4b45-70c3-4ab2-8960-cf5f56c36a04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"
Nov 02 12:08:34.890420 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.890371    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"]
Nov 02 12:08:34.951293 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.951235    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zff8\" (UniqueName: \"kubernetes.io/projected/964e4b45-70c3-4ab2-8960-cf5f56c36a04-kube-api-access-8zff8\") pod \"payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l\" (UID: \"964e4b45-70c3-4ab2-8960-cf5f56c36a04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"
Nov 02 12:08:34.991221 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.991194    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e85c3283-ce74-4e7f-b191-79e497d42a77-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h\" (UID: \"e85c3283-ce74-4e7f-b191-79e497d42a77\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"
Nov 02 12:08:34.991342 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.991244    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pn54\" (UniqueName: \"kubernetes.io/projected/e85c3283-ce74-4e7f-b191-79e497d42a77-kube-api-access-7pn54\") pod \"payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h\" (UID: \"e85c3283-ce74-4e7f-b191-79e497d42a77\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"
Nov 02 12:08:34.991856 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:34.991836    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e85c3283-ce74-4e7f-b191-79e497d42a77-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h\" (UID: \"e85c3283-ce74-4e7f-b191-79e497d42a77\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"
Nov 02 12:08:35.003060 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:35.003038    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"
Nov 02 12:08:35.056997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:35.054238    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pn54\" (UniqueName: \"kubernetes.io/projected/e85c3283-ce74-4e7f-b191-79e497d42a77-kube-api-access-7pn54\") pod \"payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h\" (UID: \"e85c3283-ce74-4e7f-b191-79e497d42a77\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"
Nov 02 12:08:35.142081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:35.142045    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"
Nov 02 12:08:35.372028 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:35.372004    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"]
Nov 02 12:08:35.373887 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:08:35.373859    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964e4b45_70c3_4ab2_8960_cf5f56c36a04.slice/crio-cff3501cecbf7d5ae55021efed8e05c5b188103b756c72377aaa7365732e5931 WatchSource:0}: Error finding container cff3501cecbf7d5ae55021efed8e05c5b188103b756c72377aaa7365732e5931: Status 404 returned error can't find the container with id cff3501cecbf7d5ae55021efed8e05c5b188103b756c72377aaa7365732e5931
Nov 02 12:08:35.511676 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:35.511644    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"]
Nov 02 12:08:35.516496 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:08:35.516472    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85c3283_ce74_4e7f_b191_79e497d42a77.slice/crio-191060eb0c99d4bdde77fa2ef5a5b3df0a2c50b8e4d57ee6c3a7da232b2e8063 WatchSource:0}: Error finding container 191060eb0c99d4bdde77fa2ef5a5b3df0a2c50b8e4d57ee6c3a7da232b2e8063: Status 404 returned error can't find the container with id 191060eb0c99d4bdde77fa2ef5a5b3df0a2c50b8e4d57ee6c3a7da232b2e8063
Nov 02 12:08:35.790754 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:35.790718    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h" event={"ID":"e85c3283-ce74-4e7f-b191-79e497d42a77","Type":"ContainerStarted","Data":"323ed9066018c5b0096dc22e3b6963b146e3f7934e16502372f0798dac762216"}
Nov 02 12:08:35.791216 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:35.790769    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h" event={"ID":"e85c3283-ce74-4e7f-b191-79e497d42a77","Type":"ContainerStarted","Data":"191060eb0c99d4bdde77fa2ef5a5b3df0a2c50b8e4d57ee6c3a7da232b2e8063"}
Nov 02 12:08:35.791216 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:35.790820    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"
Nov 02 12:08:35.792376 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:35.792355    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l" event={"ID":"964e4b45-70c3-4ab2-8960-cf5f56c36a04","Type":"ContainerStarted","Data":"db6d4de928c1321b1f6297cb9d10d4162c5fdc6f62931b10d3b758864ca4d128"}
Nov 02 12:08:35.792462 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:35.792394    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l" event={"ID":"964e4b45-70c3-4ab2-8960-cf5f56c36a04","Type":"ContainerStarted","Data":"cff3501cecbf7d5ae55021efed8e05c5b188103b756c72377aaa7365732e5931"}
Nov 02 12:08:35.792462 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:35.792414    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"
Nov 02 12:08:36.041067 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:36.032876    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h" podStartSLOduration=2.032857602 podStartE2EDuration="2.032857602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:08:35.914580054 +0000 UTC m=+744.478288463" watchObservedRunningTime="2026-11-02 12:08:36.032857602 +0000 UTC m=+744.596566012"
Nov 02 12:08:36.042586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:36.042517    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l" podStartSLOduration=2.042497297 podStartE2EDuration="2.042497297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:08:36.031817525 +0000 UTC m=+744.595526271" watchObservedRunningTime="2026-11-02 12:08:36.042497297 +0000 UTC m=+744.606205705"
Nov 02 12:08:39.598636 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.598608    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"]
Nov 02 12:08:39.601392 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.601374    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"
Nov 02 12:08:39.657557 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.657530    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-4b9210\""
Nov 02 12:08:39.709406 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.709375    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-4b9210-dockercfg-8lmhj\""
Nov 02 12:08:39.712355 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.712326    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"]
Nov 02 12:08:39.733825 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.733796    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w8zp\" (UniqueName: \"kubernetes.io/projected/42089fd9-dfb3-4681-8a92-c7b66ee2b7fb-kube-api-access-8w8zp\") pod \"payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2\" (UID: \"42089fd9-dfb3-4681-8a92-c7b66ee2b7fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"
Nov 02 12:08:39.733985 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.733848    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42089fd9-dfb3-4681-8a92-c7b66ee2b7fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2\" (UID: \"42089fd9-dfb3-4681-8a92-c7b66ee2b7fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"
Nov 02 12:08:39.754463 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.754436    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"]
Nov 02 12:08:39.757200 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.757183    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"
Nov 02 12:08:39.816949 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.816919    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"]
Nov 02 12:08:39.835301 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.835270    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-974fz\" (UniqueName: \"kubernetes.io/projected/1b98a955-089d-4f50-a719-4918f7f5a2c0-kube-api-access-974fz\") pod \"payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr\" (UID: \"1b98a955-089d-4f50-a719-4918f7f5a2c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"
Nov 02 12:08:39.835455 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.835344    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w8zp\" (UniqueName: \"kubernetes.io/projected/42089fd9-dfb3-4681-8a92-c7b66ee2b7fb-kube-api-access-8w8zp\") pod \"payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2\" (UID: \"42089fd9-dfb3-4681-8a92-c7b66ee2b7fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"
Nov 02 12:08:39.835455 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.835382    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42089fd9-dfb3-4681-8a92-c7b66ee2b7fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2\" (UID: \"42089fd9-dfb3-4681-8a92-c7b66ee2b7fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"
Nov 02 12:08:39.835455 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.835410    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98a955-089d-4f50-a719-4918f7f5a2c0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr\" (UID: \"1b98a955-089d-4f50-a719-4918f7f5a2c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"
Nov 02 12:08:39.836018 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.835997    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42089fd9-dfb3-4681-8a92-c7b66ee2b7fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2\" (UID: \"42089fd9-dfb3-4681-8a92-c7b66ee2b7fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"
Nov 02 12:08:39.900583 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.900516    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w8zp\" (UniqueName: \"kubernetes.io/projected/42089fd9-dfb3-4681-8a92-c7b66ee2b7fb-kube-api-access-8w8zp\") pod \"payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2\" (UID: \"42089fd9-dfb3-4681-8a92-c7b66ee2b7fb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"
Nov 02 12:08:39.914380 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.914360    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"
Nov 02 12:08:39.936540 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.936508    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974fz\" (UniqueName: \"kubernetes.io/projected/1b98a955-089d-4f50-a719-4918f7f5a2c0-kube-api-access-974fz\") pod \"payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr\" (UID: \"1b98a955-089d-4f50-a719-4918f7f5a2c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"
Nov 02 12:08:39.936689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.936670    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98a955-089d-4f50-a719-4918f7f5a2c0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr\" (UID: \"1b98a955-089d-4f50-a719-4918f7f5a2c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"
Nov 02 12:08:39.937531 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:39.937508    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98a955-089d-4f50-a719-4918f7f5a2c0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr\" (UID: \"1b98a955-089d-4f50-a719-4918f7f5a2c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"
Nov 02 12:08:40.025451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:40.025384    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-974fz\" (UniqueName: \"kubernetes.io/projected/1b98a955-089d-4f50-a719-4918f7f5a2c0-kube-api-access-974fz\") pod \"payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr\" (UID: \"1b98a955-089d-4f50-a719-4918f7f5a2c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"
Nov 02 12:08:40.068882 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:40.068854    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"
Nov 02 12:08:40.286703 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:40.286664    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"]
Nov 02 12:08:40.288266 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:08:40.288239    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42089fd9_dfb3_4681_8a92_c7b66ee2b7fb.slice/crio-5104ef8e61fec056a5f5507831a7218e97a88f28d02ab5518ea76c9e141bf705 WatchSource:0}: Error finding container 5104ef8e61fec056a5f5507831a7218e97a88f28d02ab5518ea76c9e141bf705: Status 404 returned error can't find the container with id 5104ef8e61fec056a5f5507831a7218e97a88f28d02ab5518ea76c9e141bf705
Nov 02 12:08:40.437438 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:40.437410    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"]
Nov 02 12:08:40.438916 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:08:40.438894    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b98a955_089d_4f50_a719_4918f7f5a2c0.slice/crio-2e1496526234d589b1ceca239c7823b1ba564a06c7496c2a295211512145764b WatchSource:0}: Error finding container 2e1496526234d589b1ceca239c7823b1ba564a06c7496c2a295211512145764b: Status 404 returned error can't find the container with id 2e1496526234d589b1ceca239c7823b1ba564a06c7496c2a295211512145764b
Nov 02 12:08:40.821358 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:40.821317    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr" event={"ID":"1b98a955-089d-4f50-a719-4918f7f5a2c0","Type":"ContainerStarted","Data":"3da45036038f9218a498bce32525296e6c6966b93c83900dcbf0cfecd79b458c"}
Nov 02 12:08:40.821819 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:40.821384    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr" event={"ID":"1b98a955-089d-4f50-a719-4918f7f5a2c0","Type":"ContainerStarted","Data":"2e1496526234d589b1ceca239c7823b1ba564a06c7496c2a295211512145764b"}
Nov 02 12:08:40.821819 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:40.821431    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"
Nov 02 12:08:40.822865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:40.822836    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2" event={"ID":"42089fd9-dfb3-4681-8a92-c7b66ee2b7fb","Type":"ContainerStarted","Data":"f5bad896d78525469477c73e3b7fcf7258160a5b69e9f3ce6bd84f29c456b6fa"}
Nov 02 12:08:40.822985 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:40.822874    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2" event={"ID":"42089fd9-dfb3-4681-8a92-c7b66ee2b7fb","Type":"ContainerStarted","Data":"5104ef8e61fec056a5f5507831a7218e97a88f28d02ab5518ea76c9e141bf705"}
Nov 02 12:08:40.822985 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:40.822962    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"
Nov 02 12:08:40.938769 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:40.938722    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr" podStartSLOduration=1.938710731 podStartE2EDuration="1.938710731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:08:40.93800181 +0000 UTC m=+749.501710223" watchObservedRunningTime="2026-11-02 12:08:40.938710731 +0000 UTC m=+749.502419140"
Nov 02 12:08:41.056648 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:41.056599    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2" podStartSLOduration=2.056585998 podStartE2EDuration="2.056585998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:08:41.054676263 +0000 UTC m=+749.618384692" watchObservedRunningTime="2026-11-02 12:08:41.056585998 +0000 UTC m=+749.620294406"
Nov 02 12:08:42.151748 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:42.151713    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"]
Nov 02 12:08:42.154815 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:42.154797    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"
Nov 02 12:08:42.214720 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:42.214687    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b6f3d8\""
Nov 02 12:08:42.214720 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:42.214719    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b6f3d8-dockercfg-7j85g\""
Nov 02 12:08:42.260945 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:42.260906    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8vjn\" (UniqueName: \"kubernetes.io/projected/18d62ecb-65b0-427f-8177-9a0fb203a358-kube-api-access-p8vjn\") pod \"payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn\" (UID: \"18d62ecb-65b0-427f-8177-9a0fb203a358\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"
Nov 02 12:08:42.261139 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:42.261017    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d62ecb-65b0-427f-8177-9a0fb203a358-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn\" (UID: \"18d62ecb-65b0-427f-8177-9a0fb203a358\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"
Nov 02 12:08:42.275785 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:42.275746    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"]
Nov 02 12:08:42.361875 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:42.361834    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8vjn\" (UniqueName: \"kubernetes.io/projected/18d62ecb-65b0-427f-8177-9a0fb203a358-kube-api-access-p8vjn\") pod \"payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn\" (UID: \"18d62ecb-65b0-427f-8177-9a0fb203a358\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"
Nov 02 12:08:42.362059 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:42.361890    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d62ecb-65b0-427f-8177-9a0fb203a358-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn\" (UID: \"18d62ecb-65b0-427f-8177-9a0fb203a358\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"
Nov 02 12:08:42.362608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:42.362588    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d62ecb-65b0-427f-8177-9a0fb203a358-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn\" (UID: \"18d62ecb-65b0-427f-8177-9a0fb203a358\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"
Nov 02 12:08:42.426652 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:42.426584    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8vjn\" (UniqueName: \"kubernetes.io/projected/18d62ecb-65b0-427f-8177-9a0fb203a358-kube-api-access-p8vjn\") pod \"payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn\" (UID: \"18d62ecb-65b0-427f-8177-9a0fb203a358\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"
Nov 02 12:08:42.468890 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:42.468853    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"
Nov 02 12:08:42.838456 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:42.838424    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"]
Nov 02 12:08:42.839451 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:08:42.839423    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d62ecb_65b0_427f_8177_9a0fb203a358.slice/crio-787517ddc0fe03541538c07f88e689d843ae416d3df1322adffba059acae548e WatchSource:0}: Error finding container 787517ddc0fe03541538c07f88e689d843ae416d3df1322adffba059acae548e: Status 404 returned error can't find the container with id 787517ddc0fe03541538c07f88e689d843ae416d3df1322adffba059acae548e
Nov 02 12:08:43.839137 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:43.839100    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn" event={"ID":"18d62ecb-65b0-427f-8177-9a0fb203a358","Type":"ContainerStarted","Data":"62742b04853465915374c6f6272e6b3d83eda72a6a1335ea332b23bd3cd52cc0"}
Nov 02 12:08:43.839591 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:43.839157    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn" event={"ID":"18d62ecb-65b0-427f-8177-9a0fb203a358","Type":"ContainerStarted","Data":"787517ddc0fe03541538c07f88e689d843ae416d3df1322adffba059acae548e"}
Nov 02 12:08:43.839591 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:43.839241    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"
Nov 02 12:08:43.962366 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:43.962316    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn" podStartSLOduration=1.9622994870000001 podStartE2EDuration="1.962299487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:08:43.960574822 +0000 UTC m=+752.524283232" watchObservedRunningTime="2026-11-02 12:08:43.962299487 +0000 UTC m=+752.526007895"
Nov 02 12:08:45.380142 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:45.380112    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:08:46.800457 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:46.800427    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"
Nov 02 12:08:46.800855 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:46.800492    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"
Nov 02 12:08:51.831321 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:51.831289    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"
Nov 02 12:08:51.831795 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:51.831374    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"
Nov 02 12:08:53.876134 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:53.876100    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:08:54.288716 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.288685    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"]
Nov 02 12:08:54.291142 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.291127    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"
Nov 02 12:08:54.349072 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.349017    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:08:54.399104 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.399078    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49886a1f-ed79-4a50-a7fc-cef24cfa5992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6798fd5-vmxnt\" (UID: \"49886a1f-ed79-4a50-a7fc-cef24cfa5992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"
Nov 02 12:08:54.399239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.399115    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49886a1f-ed79-4a50-a7fc-cef24cfa5992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6798fd5-vmxnt\" (UID: \"49886a1f-ed79-4a50-a7fc-cef24cfa5992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"
Nov 02 12:08:54.399239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.399134    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjlcp\" (UniqueName: \"kubernetes.io/projected/49886a1f-ed79-4a50-a7fc-cef24cfa5992-kube-api-access-bjlcp\") pod \"maas-api-revoke-sub-c6798fd5-vmxnt\" (UID: \"49886a1f-ed79-4a50-a7fc-cef24cfa5992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"
Nov 02 12:08:54.399760 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.399721    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:08:54.399803 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.399759    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:08:54.401899 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.401879    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"]
Nov 02 12:08:54.449270 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.449250    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:08:54.500476 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.500450    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49886a1f-ed79-4a50-a7fc-cef24cfa5992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6798fd5-vmxnt\" (UID: \"49886a1f-ed79-4a50-a7fc-cef24cfa5992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"
Nov 02 12:08:54.500615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.500487    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49886a1f-ed79-4a50-a7fc-cef24cfa5992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6798fd5-vmxnt\" (UID: \"49886a1f-ed79-4a50-a7fc-cef24cfa5992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"
Nov 02 12:08:54.500615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.500504    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjlcp\" (UniqueName: \"kubernetes.io/projected/49886a1f-ed79-4a50-a7fc-cef24cfa5992-kube-api-access-bjlcp\") pod \"maas-api-revoke-sub-c6798fd5-vmxnt\" (UID: \"49886a1f-ed79-4a50-a7fc-cef24cfa5992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"
Nov 02 12:08:54.501126 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.501107    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49886a1f-ed79-4a50-a7fc-cef24cfa5992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6798fd5-vmxnt\" (UID: \"49886a1f-ed79-4a50-a7fc-cef24cfa5992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"
Nov 02 12:08:54.502795 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.502776    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49886a1f-ed79-4a50-a7fc-cef24cfa5992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6798fd5-vmxnt\" (UID: \"49886a1f-ed79-4a50-a7fc-cef24cfa5992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"
Nov 02 12:08:54.562192 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.562169    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjlcp\" (UniqueName: \"kubernetes.io/projected/49886a1f-ed79-4a50-a7fc-cef24cfa5992-kube-api-access-bjlcp\") pod \"maas-api-revoke-sub-c6798fd5-vmxnt\" (UID: \"49886a1f-ed79-4a50-a7fc-cef24cfa5992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"
Nov 02 12:08:54.602655 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.602626    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"
Nov 02 12:08:54.848910 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.848010    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"
Nov 02 12:08:54.927675 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.927590    4464 generic.go:356] "Generic (PLEG): container finished" podID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerID="886d0c3fe1f711852cb41aa5416170ccc082175ccb5d914f176925e7349f7fcd" exitCode=0
Nov 02 12:08:54.928167 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.927667    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" event={"ID":"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7","Type":"ContainerDied","Data":"886d0c3fe1f711852cb41aa5416170ccc082175ccb5d914f176925e7349f7fcd"}
Nov 02 12:08:54.928167 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.928158    4464 scope.go:122] "RemoveContainer" containerID="886d0c3fe1f711852cb41aa5416170ccc082175ccb5d914f176925e7349f7fcd"
Nov 02 12:08:54.929387 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.929361    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:08:54.929482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.929440    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:08:54.929482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.929472    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:08:54.992199 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:54.991084    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"]
Nov 02 12:08:54.994690 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:08:54.994620    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49886a1f_ed79_4a50_a7fc_cef24cfa5992.slice/crio-e1886eb5a67ead1eec98b845e1f3eb02b425dc3960a763d54e70156da00ca6be WatchSource:0}: Error finding container e1886eb5a67ead1eec98b845e1f3eb02b425dc3960a763d54e70156da00ca6be: Status 404 returned error can't find the container with id e1886eb5a67ead1eec98b845e1f3eb02b425dc3960a763d54e70156da00ca6be
Nov 02 12:08:55.935564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:55.935528    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" event={"ID":"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7","Type":"ContainerStarted","Data":"1d395adc2e1c2293dd782593366d6e57e6dec93917a756c5734c605d732501e1"}
Nov 02 12:08:55.936841 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:55.936817    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt" event={"ID":"49886a1f-ed79-4a50-a7fc-cef24cfa5992","Type":"ContainerStarted","Data":"e1886eb5a67ead1eec98b845e1f3eb02b425dc3960a763d54e70156da00ca6be"}
Nov 02 12:08:56.413207 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:56.413155    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:56.414263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:56.414236    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:08:56.414336 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:56.414312    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 12:08:57.413756 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:57.413309    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:08:57.413756 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:57.413383    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 12:08:58.339296 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:08:58.339260    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:08:58.413287 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:58.413219    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:08:58.413287 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:58.413273    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 12:08:58.750098 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:58.750052    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:08:59.413831 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:59.413600    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:08:59.413831 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:08:59.413652    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 12:09:00.413736 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:00.413706    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:09:00.414162 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:00.413757    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 12:09:01.006698 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:01.006661    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt" event={"ID":"49886a1f-ed79-4a50-a7fc-cef24cfa5992","Type":"ContainerStarted","Data":"c5efec9d4b5724f2c515ac355134f30335285eaf45bd70a94af64110e04ef2ca"}
Nov 02 12:09:01.142843 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:01.142782    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt" podStartSLOduration=2.098187877 podStartE2EDuration="7.142764708s" totalImagesPullingTime="5.044576831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:08:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:09:01.140383461 +0000 UTC m=+769.704091871" watchObservedRunningTime="2026-11-02 12:09:01.142764708 +0000 UTC m=+769.706473118"
Nov 02 12:09:01.413460 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:01.413369    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:09:01.413460 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:01.413426    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 02 12:09:02.014142 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:02.014102    4464 generic.go:356] "Generic (PLEG): container finished" podID="49886a1f-ed79-4a50-a7fc-cef24cfa5992" containerID="c5efec9d4b5724f2c515ac355134f30335285eaf45bd70a94af64110e04ef2ca" exitCode=0
Nov 02 12:09:02.014697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:02.014188    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt" event={"ID":"49886a1f-ed79-4a50-a7fc-cef24cfa5992","Type":"ContainerDied","Data":"c5efec9d4b5724f2c515ac355134f30335285eaf45bd70a94af64110e04ef2ca"}
Nov 02 12:09:02.417257 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:02.417180    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:09:02.418302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:02.418279    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:09:03.185807 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:03.185780    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"
Nov 02 12:09:03.343854 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:03.343774    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49886a1f-ed79-4a50-a7fc-cef24cfa5992-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49886a1f-ed79-4a50-a7fc-cef24cfa5992-maas-api-openshift-ca\") pod \"49886a1f-ed79-4a50-a7fc-cef24cfa5992\" (UID: \"49886a1f-ed79-4a50-a7fc-cef24cfa5992\") "
Nov 02 12:09:03.344018 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:03.343847    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49886a1f-ed79-4a50-a7fc-cef24cfa5992-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49886a1f-ed79-4a50-a7fc-cef24cfa5992-maas-api-xks-ca\") pod \"49886a1f-ed79-4a50-a7fc-cef24cfa5992\" (UID: \"49886a1f-ed79-4a50-a7fc-cef24cfa5992\") "
Nov 02 12:09:03.344018 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:03.343919    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49886a1f-ed79-4a50-a7fc-cef24cfa5992-kube-api-access-bjlcp\" (UniqueName: \"kubernetes.io/projected/49886a1f-ed79-4a50-a7fc-cef24cfa5992-kube-api-access-bjlcp\") pod \"49886a1f-ed79-4a50-a7fc-cef24cfa5992\" (UID: \"49886a1f-ed79-4a50-a7fc-cef24cfa5992\") "
Nov 02 12:09:03.344233 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:03.344207    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49886a1f-ed79-4a50-a7fc-cef24cfa5992-maas-api-openshift-ca" pod "49886a1f-ed79-4a50-a7fc-cef24cfa5992" (UID: "49886a1f-ed79-4a50-a7fc-cef24cfa5992"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:09:03.344360 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:03.344340    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49886a1f-ed79-4a50-a7fc-cef24cfa5992-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:09:03.346953 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:03.346930    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49886a1f-ed79-4a50-a7fc-cef24cfa5992-maas-api-xks-ca" pod "49886a1f-ed79-4a50-a7fc-cef24cfa5992" (UID: "49886a1f-ed79-4a50-a7fc-cef24cfa5992"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:09:03.348049 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:03.348004    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49886a1f-ed79-4a50-a7fc-cef24cfa5992-kube-api-access-bjlcp" pod "49886a1f-ed79-4a50-a7fc-cef24cfa5992" (UID: "49886a1f-ed79-4a50-a7fc-cef24cfa5992"). InnerVolumeSpecName "kube-api-access-bjlcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:09:03.445526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:03.445486    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49886a1f-ed79-4a50-a7fc-cef24cfa5992-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:09:03.445526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:03.445511    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjlcp\" (UniqueName: \"kubernetes.io/projected/49886a1f-ed79-4a50-a7fc-cef24cfa5992-kube-api-access-bjlcp\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:09:03.943661 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:09:03.943617    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739b7556_be7f_4208_b953_4f82f7511f3e.slice/crio-a5eddea6d73d2504c8942f3c73e7ac577b68931e325a29cd44dedc1bb580f712.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739b7556_be7f_4208_b953_4f82f7511f3e.slice/crio-conmon-a5eddea6d73d2504c8942f3c73e7ac577b68931e325a29cd44dedc1bb580f712.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:09:04.042473 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:04.042399    4464 generic.go:356] "Generic (PLEG): container finished" podID="739b7556-be7f-4208-b953-4f82f7511f3e" containerID="a5eddea6d73d2504c8942f3c73e7ac577b68931e325a29cd44dedc1bb580f712" exitCode=0
Nov 02 12:09:04.042634 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:04.042478    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9" event={"ID":"739b7556-be7f-4208-b953-4f82f7511f3e","Type":"ContainerDied","Data":"a5eddea6d73d2504c8942f3c73e7ac577b68931e325a29cd44dedc1bb580f712"}
Nov 02 12:09:04.042958 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:04.042932    4464 scope.go:122] "RemoveContainer" containerID="a5eddea6d73d2504c8942f3c73e7ac577b68931e325a29cd44dedc1bb580f712"
Nov 02 12:09:04.043946 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:04.043922    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt" event={"ID":"49886a1f-ed79-4a50-a7fc-cef24cfa5992","Type":"ContainerDied","Data":"e1886eb5a67ead1eec98b845e1f3eb02b425dc3960a763d54e70156da00ca6be"}
Nov 02 12:09:04.044068 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:04.043965    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1886eb5a67ead1eec98b845e1f3eb02b425dc3960a763d54e70156da00ca6be"
Nov 02 12:09:04.044068 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:04.043987    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"
Nov 02 12:09:04.044179 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:04.044146    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:09:04.044231 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:04.044205    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:09:04.044286 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:04.044236    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:09:05.051347 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:05.051309    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9" event={"ID":"739b7556-be7f-4208-b953-4f82f7511f3e","Type":"ContainerStarted","Data":"4b5a4d1d84cb505cd351a8dab6c80d886b2330ea3de36739490c3a13933c0f63"}
Nov 02 12:09:05.711743 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:05.703452    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:09:05.711743 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:05.703586    4464 patch_prober.go:29] interesting pod/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 12:09:05.711743 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:05.703635    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9" podUID="739b7556-be7f-4208-b953-4f82f7511f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 12:09:06.700012 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:06.699432    4464 patch_prober.go:29] interesting pod/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 12:09:06.700012 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:06.699492    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9" podUID="739b7556-be7f-4208-b953-4f82f7511f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 12:09:07.699292 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:07.699260    4464 patch_prober.go:29] interesting pod/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 12:09:07.699441 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:07.699313    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9" podUID="739b7556-be7f-4208-b953-4f82f7511f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 12:09:08.699288 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:08.699257    4464 patch_prober.go:29] interesting pod/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 12:09:08.699698 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:08.699309    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9" podUID="739b7556-be7f-4208-b953-4f82f7511f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 12:09:09.698723 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:09.698685    4464 patch_prober.go:29] interesting pod/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 12:09:09.698883 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:09.698751    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9" podUID="739b7556-be7f-4208-b953-4f82f7511f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 12:09:10.704409 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:10.704370    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:09:10.704829 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:10.704758    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:09:10.705513 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:10.705490    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:09:13.671103 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.671070    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"]
Nov 02 12:09:13.671549 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.671536    4464 cpu_manager.go:414] "containerMap: removing container" podUID="49886a1f-ed79-4a50-a7fc-cef24cfa5992" containerName="revoke-keys"
Nov 02 12:09:13.671549 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.671549    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49886a1f-ed79-4a50-a7fc-cef24cfa5992" containerName="revoke-keys"
Nov 02 12:09:13.671645 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.671635    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="49886a1f-ed79-4a50-a7fc-cef24cfa5992" containerName="revoke-keys"
Nov 02 12:09:13.674223 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.674206    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.685735 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.685709    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-data\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.685824 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.685747    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-credential-socket\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.685824 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.685767    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istiod-ca-cert\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.685917 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.685890    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-workload-certs\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.685954 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.685942    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-token\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.686033 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.686018    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zct88\" (UniqueName: \"kubernetes.io/projected/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-kube-api-access-zct88\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.686075 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.686040    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-envoy\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.686126 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.686073    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-workload-socket\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.686181 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.686146    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-podinfo\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.728176 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.728140    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-966817-openshift-default-dockercfg-ksb5h\""
Nov 02 12:09:13.781633 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.781596    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"]
Nov 02 12:09:13.792709 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.792678    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-credential-socket\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.792886 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.792720    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istiod-ca-cert\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.792886 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.792769    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-workload-certs\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.792886 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.792831    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-token\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.792886 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.792857    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zct88\" (UniqueName: \"kubernetes.io/projected/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-kube-api-access-zct88\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.792886 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.792885    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-envoy\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.793184 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.792945    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-workload-socket\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.793184 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.792986    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-podinfo\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.793184 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.793077    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-data\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.793526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.793506    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-data\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.793621 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.793596    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-credential-socket\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.794428 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.794400    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istiod-ca-cert\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.794648 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.794624    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-workload-socket\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.795152 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.795027    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-workload-certs\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.796762 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.796733    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-podinfo\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.797848 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.797824    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-envoy\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.859179 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.859130    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zct88\" (UniqueName: \"kubernetes.io/projected/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-kube-api-access-zct88\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.865244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.864532    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-token\") pod \"e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") " pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:13.985931 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:13.985900    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:14.412215 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.412141    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"]
Nov 02 12:09:14.415610 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.415588    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.426141 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.426115    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"]
Nov 02 12:09:14.427638 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:09:14.427599    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3f68ba_d0bc_4ae1_b9c1_88850db07172.slice/crio-1abaad349b6bec4b9eb3988cab2b8a2d2cd420e59f0bbfdc5f8b999047ba3786 WatchSource:0}: Error finding container 1abaad349b6bec4b9eb3988cab2b8a2d2cd420e59f0bbfdc5f8b999047ba3786: Status 404 returned error can't find the container with id 1abaad349b6bec4b9eb3988cab2b8a2d2cd420e59f0bbfdc5f8b999047ba3786
Nov 02 12:09:14.430197 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.430163    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:09:14.430289 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.430239    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:09:14.430289 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.430283    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:09:14.478793 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.478769    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"]
Nov 02 12:09:14.498161 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.498134    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.498279 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.498168    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.498279 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.498189    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.498354 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.498307    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.498354 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.498345    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.498443 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.498434    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psqxd\" (UniqueName: \"kubernetes.io/projected/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-kube-api-access-psqxd\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.498483 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.498463    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.498543 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.498503    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.498604 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.498592    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.599304 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.599267    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psqxd\" (UniqueName: \"kubernetes.io/projected/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-kube-api-access-psqxd\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.599485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.599317    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.599485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.599362    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.599485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.599395    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.599663 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.599546    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.599663 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.599592    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.599663 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.599624    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.599832 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.599676    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.599832 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.599726    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.599936 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.599836    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.600061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.600013    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.600061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.600045    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.600239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.600185    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.600482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.600458    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.601908 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.601885    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.602079 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.602063    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.661491 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.661462    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.662124 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.662094    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psqxd\" (UniqueName: \"kubernetes.io/projected/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-kube-api-access-psqxd\") pod \"maas-default-gateway-openshift-default-6dcf757899-mpqgm\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:14.729998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:14.729952    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:15.128641 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:15.128538    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5" event={"ID":"ad3f68ba-d0bc-4ae1-b9c1-88850db07172","Type":"ContainerStarted","Data":"6dba20abb675323d063d4d97e321c1a3a50116afbd189fd3fe03005338162c9d"}
Nov 02 12:09:15.128641 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:15.128603    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5" event={"ID":"ad3f68ba-d0bc-4ae1-b9c1-88850db07172","Type":"ContainerStarted","Data":"1abaad349b6bec4b9eb3988cab2b8a2d2cd420e59f0bbfdc5f8b999047ba3786"}
Nov 02 12:09:15.153835 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:15.153813    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"]
Nov 02 12:09:15.155448 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:09:15.155419    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36f100c_b555_4ffd_b2d3_d3ffbbb69e25.slice/crio-04c77138974af64140d8fe74604a9108e16d7238f75a843592a56f2247e82041 WatchSource:0}: Error finding container 04c77138974af64140d8fe74604a9108e16d7238f75a843592a56f2247e82041: Status 404 returned error can't find the container with id 04c77138974af64140d8fe74604a9108e16d7238f75a843592a56f2247e82041
Nov 02 12:09:15.157732 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:15.157703    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:09:15.157827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:15.157786    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:09:15.157889 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:15.157833    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:09:15.256886 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:15.256824    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5" podStartSLOduration=2.256805945 podStartE2EDuration="2.256805945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:09:15.252823288 +0000 UTC m=+783.816531698" watchObservedRunningTime="2026-11-02 12:09:15.256805945 +0000 UTC m=+783.820514357"
Nov 02 12:09:15.994999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:15.994366    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:16.011457 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:16.011195    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:16.135939 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:16.135901    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm" event={"ID":"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25","Type":"ContainerStarted","Data":"ef10021c51e6b358bc480b49b9aa15e60f62df94bad10212eadbdeb94ea3686f"}
Nov 02 12:09:16.136129 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:16.135958    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm" event={"ID":"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25","Type":"ContainerStarted","Data":"04c77138974af64140d8fe74604a9108e16d7238f75a843592a56f2247e82041"}
Nov 02 12:09:16.136183 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:16.136139    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:16.137076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:16.137059    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:16.283055 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:16.282565    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm" podStartSLOduration=2.282541126 podStartE2EDuration="2.282541126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:09:16.273385145 +0000 UTC m=+784.837093567" watchObservedRunningTime="2026-11-02 12:09:16.282541126 +0000 UTC m=+784.846249525"
Nov 02 12:09:16.730323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:16.730289    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:16.731866 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:16.731840    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mpqgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 12:09:16.732027 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:16.731898    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm" podUID="d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 12:09:16.858454 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:16.858421    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:09:17.731182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:17.731144    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mpqgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 12:09:17.731588 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:17.731204    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm" podUID="d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 12:09:18.730281 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:18.730247    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mpqgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 12:09:18.730456 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:18.730306    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm" podUID="d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 12:09:19.730636 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:19.730598    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mpqgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 12:09:19.731065 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:19.730658    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm" podUID="d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 12:09:20.730909 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:20.730871    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mpqgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 12:09:20.731312 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:20.730923    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm" podUID="d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 12:09:21.730499 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:21.730467    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mpqgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 12:09:21.730692 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:21.730521    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm" podUID="d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 12:09:21.943256 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:21.943226    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"]
Nov 02 12:09:21.946093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:21.946067    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"
Nov 02 12:09:22.001885 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.001792    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-966817\""
Nov 02 12:09:22.013351 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.013322    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcvp6\" (UniqueName: \"kubernetes.io/projected/e0675eaf-e7c8-4516-bc84-9796fcdd505d-kube-api-access-hcvp6\") pod \"payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h\" (UID: \"e0675eaf-e7c8-4516-bc84-9796fcdd505d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"
Nov 02 12:09:22.013565 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.013521    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0675eaf-e7c8-4516-bc84-9796fcdd505d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h\" (UID: \"e0675eaf-e7c8-4516-bc84-9796fcdd505d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"
Nov 02 12:09:22.051760 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.051486    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-966817-dockercfg-zn542\""
Nov 02 12:09:22.055161 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.055133    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"]
Nov 02 12:09:22.106655 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.106615    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"]
Nov 02 12:09:22.109671 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.109648    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"
Nov 02 12:09:22.114946 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.114601    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcvp6\" (UniqueName: \"kubernetes.io/projected/e0675eaf-e7c8-4516-bc84-9796fcdd505d-kube-api-access-hcvp6\") pod \"payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h\" (UID: \"e0675eaf-e7c8-4516-bc84-9796fcdd505d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"
Nov 02 12:09:22.114946 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.114786    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0675eaf-e7c8-4516-bc84-9796fcdd505d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h\" (UID: \"e0675eaf-e7c8-4516-bc84-9796fcdd505d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"
Nov 02 12:09:22.115726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.115692    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0675eaf-e7c8-4516-bc84-9796fcdd505d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h\" (UID: \"e0675eaf-e7c8-4516-bc84-9796fcdd505d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"
Nov 02 12:09:22.140870 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.140839    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"]
Nov 02 12:09:22.178701 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.178663    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcvp6\" (UniqueName: \"kubernetes.io/projected/e0675eaf-e7c8-4516-bc84-9796fcdd505d-kube-api-access-hcvp6\") pod \"payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h\" (UID: \"e0675eaf-e7c8-4516-bc84-9796fcdd505d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"
Nov 02 12:09:22.215475 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.215441    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d832815-bc49-464b-808c-70a5116c2fed-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx\" (UID: \"4d832815-bc49-464b-808c-70a5116c2fed\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"
Nov 02 12:09:22.215580 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.215491    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxfgk\" (UniqueName: \"kubernetes.io/projected/4d832815-bc49-464b-808c-70a5116c2fed-kube-api-access-mxfgk\") pod \"payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx\" (UID: \"4d832815-bc49-464b-808c-70a5116c2fed\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"
Nov 02 12:09:22.270860 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.270794    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"
Nov 02 12:09:22.318698 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.317232    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d832815-bc49-464b-808c-70a5116c2fed-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx\" (UID: \"4d832815-bc49-464b-808c-70a5116c2fed\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"
Nov 02 12:09:22.318698 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.317323    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxfgk\" (UniqueName: \"kubernetes.io/projected/4d832815-bc49-464b-808c-70a5116c2fed-kube-api-access-mxfgk\") pod \"payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx\" (UID: \"4d832815-bc49-464b-808c-70a5116c2fed\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"
Nov 02 12:09:22.318698 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.318143    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d832815-bc49-464b-808c-70a5116c2fed-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx\" (UID: \"4d832815-bc49-464b-808c-70a5116c2fed\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"
Nov 02 12:09:22.381717 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.381150    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxfgk\" (UniqueName: \"kubernetes.io/projected/4d832815-bc49-464b-808c-70a5116c2fed-kube-api-access-mxfgk\") pod \"payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx\" (UID: \"4d832815-bc49-464b-808c-70a5116c2fed\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"
Nov 02 12:09:22.422953 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.422923    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"
Nov 02 12:09:22.645612 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.645529    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"]
Nov 02 12:09:22.650118 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:09:22.650090    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0675eaf_e7c8_4516_bc84_9796fcdd505d.slice/crio-f2c673e1686b0a13ea6025d21ca86ea2832c28f519cd7018961c8f93cb40debd WatchSource:0}: Error finding container f2c673e1686b0a13ea6025d21ca86ea2832c28f519cd7018961c8f93cb40debd: Status 404 returned error can't find the container with id f2c673e1686b0a13ea6025d21ca86ea2832c28f519cd7018961c8f93cb40debd
Nov 02 12:09:22.730303 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.730276    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mpqgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 02 12:09:22.730388 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.730320    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm" podUID="d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 02 12:09:22.795807 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:22.795782    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"]
Nov 02 12:09:22.797341 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:09:22.797310    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d832815_bc49_464b_808c_70a5116c2fed.slice/crio-b868bbc10d80a5d76d577af98d0fc834d53adac9193408bb0f123803a87c4716 WatchSource:0}: Error finding container b868bbc10d80a5d76d577af98d0fc834d53adac9193408bb0f123803a87c4716: Status 404 returned error can't find the container with id b868bbc10d80a5d76d577af98d0fc834d53adac9193408bb0f123803a87c4716
Nov 02 12:09:23.178488 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:23.178368    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h" event={"ID":"e0675eaf-e7c8-4516-bc84-9796fcdd505d","Type":"ContainerStarted","Data":"a3f17dfab3b721d5ef97caa042d3643241e5889df564c6498f1989ecc1835bc9"}
Nov 02 12:09:23.178488 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:23.178433    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h" event={"ID":"e0675eaf-e7c8-4516-bc84-9796fcdd505d","Type":"ContainerStarted","Data":"f2c673e1686b0a13ea6025d21ca86ea2832c28f519cd7018961c8f93cb40debd"}
Nov 02 12:09:23.179176 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:23.178623    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"
Nov 02 12:09:23.180227 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:23.180205    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx" event={"ID":"4d832815-bc49-464b-808c-70a5116c2fed","Type":"ContainerStarted","Data":"50e9586c153055ae1f10af17276af5e4411d49172effeba2517cb82dafc9e268"}
Nov 02 12:09:23.180351 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:23.180246    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx" event={"ID":"4d832815-bc49-464b-808c-70a5116c2fed","Type":"ContainerStarted","Data":"b868bbc10d80a5d76d577af98d0fc834d53adac9193408bb0f123803a87c4716"}
Nov 02 12:09:23.180351 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:23.180298    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"
Nov 02 12:09:23.299141 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:23.299086    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h" podStartSLOduration=2.2990703359999998 podStartE2EDuration="2.299070336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:09:23.298858889 +0000 UTC m=+791.862567299" watchObservedRunningTime="2026-11-02 12:09:23.299070336 +0000 UTC m=+791.862778747"
Nov 02 12:09:24.650586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:24.650555    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:24.651187 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:24.651008    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:24.652114 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:24.652090    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:09:24.829339 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:24.829275    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx" podStartSLOduration=2.829260931 podStartE2EDuration="2.829260931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:09:23.42318752 +0000 UTC m=+791.986895931" watchObservedRunningTime="2026-11-02 12:09:24.829260931 +0000 UTC m=+793.392969376"
Nov 02 12:09:27.073584 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:27.073552    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"]
Nov 02 12:09:27.076399 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:27.076378    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"
Nov 02 12:09:27.131123 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:27.131093    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"]
Nov 02 12:09:27.172506 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:27.172471    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc81545d-7d40-4b73-9e60-69d2348cfddd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r\" (UID: \"fc81545d-7d40-4b73-9e60-69d2348cfddd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"
Nov 02 12:09:27.172649 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:27.172588    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9xk4\" (UniqueName: \"kubernetes.io/projected/fc81545d-7d40-4b73-9e60-69d2348cfddd-kube-api-access-z9xk4\") pod \"payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r\" (UID: \"fc81545d-7d40-4b73-9e60-69d2348cfddd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"
Nov 02 12:09:27.177964 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:27.177943    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-dd087a\""
Nov 02 12:09:27.178097 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:27.178083    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-dd087a-dockercfg-nmd9w\""
Nov 02 12:09:27.273702 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:27.273670    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc81545d-7d40-4b73-9e60-69d2348cfddd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r\" (UID: \"fc81545d-7d40-4b73-9e60-69d2348cfddd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"
Nov 02 12:09:27.273868 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:27.273734    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9xk4\" (UniqueName: \"kubernetes.io/projected/fc81545d-7d40-4b73-9e60-69d2348cfddd-kube-api-access-z9xk4\") pod \"payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r\" (UID: \"fc81545d-7d40-4b73-9e60-69d2348cfddd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"
Nov 02 12:09:27.274438 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:27.274414    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc81545d-7d40-4b73-9e60-69d2348cfddd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r\" (UID: \"fc81545d-7d40-4b73-9e60-69d2348cfddd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"
Nov 02 12:09:27.336349 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:27.336289    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9xk4\" (UniqueName: \"kubernetes.io/projected/fc81545d-7d40-4b73-9e60-69d2348cfddd-kube-api-access-z9xk4\") pod \"payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r\" (UID: \"fc81545d-7d40-4b73-9e60-69d2348cfddd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"
Nov 02 12:09:27.388634 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:27.388612    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"
Nov 02 12:09:27.765055 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:27.765028    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"]
Nov 02 12:09:27.766269 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:09:27.766244    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc81545d_7d40_4b73_9e60_69d2348cfddd.slice/crio-1805afc61f25c045449553247c7f46a1c20f8ebe34e01293fe78d4597a033540 WatchSource:0}: Error finding container 1805afc61f25c045449553247c7f46a1c20f8ebe34e01293fe78d4597a033540: Status 404 returned error can't find the container with id 1805afc61f25c045449553247c7f46a1c20f8ebe34e01293fe78d4597a033540
Nov 02 12:09:28.215264 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:28.215181    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r" event={"ID":"fc81545d-7d40-4b73-9e60-69d2348cfddd","Type":"ContainerStarted","Data":"afb4c368ae4bf0971015cca0c47994b11707388ec7b58fa35d6c8b5f746c2bbd"}
Nov 02 12:09:28.215264 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:28.215233    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r" event={"ID":"fc81545d-7d40-4b73-9e60-69d2348cfddd","Type":"ContainerStarted","Data":"1805afc61f25c045449553247c7f46a1c20f8ebe34e01293fe78d4597a033540"}
Nov 02 12:09:28.215652 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:28.215289    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"
Nov 02 12:09:28.291695 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:09:28.291660    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:09:28.334257 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:28.334202    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r" podStartSLOduration=1.334185169 podStartE2EDuration="1.334185169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:09:28.332549719 +0000 UTC m=+796.896258129" watchObservedRunningTime="2026-11-02 12:09:28.334185169 +0000 UTC m=+796.897893578"
Nov 02 12:09:34.188946 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:34.188917    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"
Nov 02 12:09:34.189560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:34.189002    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"
Nov 02 12:09:39.224611 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:39.224583    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"
Nov 02 12:09:40.061187 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:40.061154    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:09:58.291376 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:09:58.291283    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:09:58.436090 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:58.436059    4464 generic.go:356] "Generic (PLEG): container finished" podID="ad3f68ba-d0bc-4ae1-b9c1-88850db07172" containerID="6dba20abb675323d063d4d97e321c1a3a50116afbd189fd3fe03005338162c9d" exitCode=0
Nov 02 12:09:58.436259 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:58.436133    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5" event={"ID":"ad3f68ba-d0bc-4ae1-b9c1-88850db07172","Type":"ContainerDied","Data":"6dba20abb675323d063d4d97e321c1a3a50116afbd189fd3fe03005338162c9d"}
Nov 02 12:09:58.436482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:58.436468    4464 scope.go:122] "RemoveContainer" containerID="6dba20abb675323d063d4d97e321c1a3a50116afbd189fd3fe03005338162c9d"
Nov 02 12:09:58.437743 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:58.437719    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:09:58.437807 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:58.437782    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:09:58.437846 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:58.437813    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:09:59.203596 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:59.203559    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:09:59.463910 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:59.463822    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5" event={"ID":"ad3f68ba-d0bc-4ae1-b9c1-88850db07172","Type":"ContainerStarted","Data":"6f159818951a10416cd26ada0a95ab9d266f46f834f115b6e2a76ad97a966159"}
Nov 02 12:09:59.986281 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:59.986221    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:09:59.987595 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:59.987569    4464 patch_prober.go:29] interesting pod/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 02 12:09:59.987722 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:09:59.987623    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5" podUID="ad3f68ba-d0bc-4ae1-b9c1-88850db07172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 02 12:10:00.987035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:00.987003    4464 patch_prober.go:29] interesting pod/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 02 12:10:00.987435 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:00.987065    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5" podUID="ad3f68ba-d0bc-4ae1-b9c1-88850db07172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 02 12:10:01.136875 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:01.136845    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:10:01.987196 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:01.987160    4464 patch_prober.go:29] interesting pod/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 02 12:10:01.987593 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:01.987216    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5" podUID="ad3f68ba-d0bc-4ae1-b9c1-88850db07172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 02 12:10:02.986724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:02.986689    4464 patch_prober.go:29] interesting pod/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 02 12:10:02.986899 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:02.986747    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5" podUID="ad3f68ba-d0bc-4ae1-b9c1-88850db07172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 02 12:10:03.986404 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:03.986370    4464 patch_prober.go:29] interesting pod/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 02 12:10:03.986903 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:03.986433    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5" podUID="ad3f68ba-d0bc-4ae1-b9c1-88850db07172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 02 12:10:04.990950 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:04.990920    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:10:04.992080 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:04.992054    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:10:12.263093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.263055    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"]
Nov 02 12:10:12.267283 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.267259    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.324805 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.324774    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"]
Nov 02 12:10:12.324964 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.324936    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b1703c\"/\"openshift-service-ca.crt\""
Nov 02 12:10:12.372343 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.372318    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b1703c\"/\"kube-root-ca.crt\""
Nov 02 12:10:12.372542 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.372464    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b1703c\"/\"default-dockercfg-ptjkv\""
Nov 02 12:10:12.372542 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.372510    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b1703c\"/\"premium-w0-b1703c-kserve-self-signed-certs\""
Nov 02 12:10:12.418463 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.418416    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-dshm\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.419056 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.419019    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4rc5\" (UniqueName: \"kubernetes.io/projected/d30985d1-0ddd-486f-8104-94f3bdb4c049-kube-api-access-v4rc5\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.419191 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.419124    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d30985d1-0ddd-486f-8104-94f3bdb4c049-tls-certs\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.419191 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.419161    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-tmp-dir\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.419306 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.419257    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-model-cache\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.419306 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.419283    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-home\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.507226 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.507195    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:10:12.520207 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.520142    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-dshm\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.520207 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.520178    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rc5\" (UniqueName: \"kubernetes.io/projected/d30985d1-0ddd-486f-8104-94f3bdb4c049-kube-api-access-v4rc5\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.520398 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.520307    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d30985d1-0ddd-486f-8104-94f3bdb4c049-tls-certs\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.520398 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.520346    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-tmp-dir\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.520510 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.520433    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-model-cache\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.520510 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.520461    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-home\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.520780 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.520744    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-tmp-dir\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.520780 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.520776    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-model-cache\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.520928 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.520800    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-home\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.522561 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.522531    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-dshm\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.522843 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.522796    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d30985d1-0ddd-486f-8104-94f3bdb4c049-tls-certs\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.582519 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.582483    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4rc5\" (UniqueName: \"kubernetes.io/projected/d30985d1-0ddd-486f-8104-94f3bdb4c049-kube-api-access-v4rc5\") pod \"premium-w0-b1703c-kserve-7988b5bcd6-8lgcj\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") " pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.589136 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.589110    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:12.961114 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:12.961090    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"]
Nov 02 12:10:12.963775 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:10:12.963749    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd30985d1_0ddd_486f_8104_94f3bdb4c049.slice/crio-17b2170db13cf723d0ef82ed6bf7541a6e0babdded1f6df2159ec5e4e96d0718 WatchSource:0}: Error finding container 17b2170db13cf723d0ef82ed6bf7541a6e0babdded1f6df2159ec5e4e96d0718: Status 404 returned error can't find the container with id 17b2170db13cf723d0ef82ed6bf7541a6e0babdded1f6df2159ec5e4e96d0718
Nov 02 12:10:13.554738 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:13.554704    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj" event={"ID":"d30985d1-0ddd-486f-8104-94f3bdb4c049","Type":"ContainerStarted","Data":"59268ddc007b5d35d67aa7331714851578cdaf3ee418a3792af697c1d2686464"}
Nov 02 12:10:13.555135 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:13.554754    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj" event={"ID":"d30985d1-0ddd-486f-8104-94f3bdb4c049","Type":"ContainerStarted","Data":"17b2170db13cf723d0ef82ed6bf7541a6e0babdded1f6df2159ec5e4e96d0718"}
Nov 02 12:10:13.614405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:13.614375    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:13.682270 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:13.682210    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj" podStartSLOduration=1.68219345 podStartE2EDuration="1.68219345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:10:13.679297153 +0000 UTC m=+842.243005563" watchObservedRunningTime="2026-11-02 12:10:13.68219345 +0000 UTC m=+842.245901853"
Nov 02 12:10:22.590143 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:22.590106    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:22.602488 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:22.602462    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:22.613474 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:22.613455    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:10:26.794999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:26.794911    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"]
Nov 02 12:10:26.799137 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:26.799117    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:26.860996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:26.854368    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-03c4e8-openshift-default-dockercfg-8bvrq\""
Nov 02 12:10:26.921207 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:26.921176    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"]
Nov 02 12:10:26.981287 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:26.981252    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-data\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:26.981287 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:26.981286    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-podinfo\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:26.981512 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:26.981364    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-envoy\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:26.981512 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:26.981402    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-workload-certs\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:26.981512 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:26.981455    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-token\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:26.981669 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:26.981534    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkz78\" (UniqueName: \"kubernetes.io/projected/9d39d963-d570-44cc-96fd-a4efee1e45fc-kube-api-access-nkz78\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:26.981669 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:26.981555    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-credential-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:26.981669 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:26.981582    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-workload-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:26.981828 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:26.981687    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d39d963-d570-44cc-96fd-a4efee1e45fc-istiod-ca-cert\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.083090 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.083011    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-data\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.083090 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.083046    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-podinfo\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.083090 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.083073    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-envoy\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.083090 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.083091    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-workload-certs\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.083419 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.083138    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-token\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.083419 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.083168    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkz78\" (UniqueName: \"kubernetes.io/projected/9d39d963-d570-44cc-96fd-a4efee1e45fc-kube-api-access-nkz78\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.083419 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.083316    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-credential-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.083419 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.083368    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-workload-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.083619 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.083437    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d39d963-d570-44cc-96fd-a4efee1e45fc-istiod-ca-cert\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.083619 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.083488    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-data\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.083826 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.083802    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-credential-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.083910 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.083838    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-workload-certs\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.083987 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.083950    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-workload-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.084143 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.084126    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d39d963-d570-44cc-96fd-a4efee1e45fc-istiod-ca-cert\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.085460 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.085430    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-envoy\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.085661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.085644    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-podinfo\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.161959 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.161923    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-token\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.163483 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.163451    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkz78\" (UniqueName: \"kubernetes.io/projected/9d39d963-d570-44cc-96fd-a4efee1e45fc-kube-api-access-nkz78\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.204409 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.204384    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:27.632738 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.632713    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"]
Nov 02 12:10:27.633632 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:10:27.633607    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d39d963_d570_44cc_96fd_a4efee1e45fc.slice/crio-ef7e5983ee81c3e586cec5d61841b85023c7150b6e57221c51a9713af3917a55 WatchSource:0}: Error finding container ef7e5983ee81c3e586cec5d61841b85023c7150b6e57221c51a9713af3917a55: Status 404 returned error can't find the container with id ef7e5983ee81c3e586cec5d61841b85023c7150b6e57221c51a9713af3917a55
Nov 02 12:10:27.635878 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.635852    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:10:27.635934 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.635911    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:10:27.635969 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:27.635938    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:10:28.291719 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:10:28.291685    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:10:28.634061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:28.633939    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d" event={"ID":"9d39d963-d570-44cc-96fd-a4efee1e45fc","Type":"ContainerStarted","Data":"b5870258acae6a70a0cdf5fc7482cf5c13d061212278229a91117c59090dbf4f"}
Nov 02 12:10:28.634291 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:28.634261    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d" event={"ID":"9d39d963-d570-44cc-96fd-a4efee1e45fc","Type":"ContainerStarted","Data":"ef7e5983ee81c3e586cec5d61841b85023c7150b6e57221c51a9713af3917a55"}
Nov 02 12:10:28.808572 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:28.808517    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d" podStartSLOduration=2.808505295 podStartE2EDuration="2.808505295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:10:28.806429512 +0000 UTC m=+857.370137919" watchObservedRunningTime="2026-11-02 12:10:28.808505295 +0000 UTC m=+857.372213703"
Nov 02 12:10:29.205384 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:29.205350    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:29.210367 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:29.210345    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:29.661334 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:29.661294    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:29.662412 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:29.662386    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:10:34.810632 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:34.810594    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:10:35.438570 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:35.438525    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"]
Nov 02 12:10:35.449927 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:35.449898    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"
Nov 02 12:10:35.495627 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:35.495596    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"]
Nov 02 12:10:35.543180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:35.543154    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-03c4e8\""
Nov 02 12:10:35.543322 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:35.543264    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-03c4e8-dockercfg-nd29c\""
Nov 02 12:10:35.614421 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:35.614396    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjv58\" (UniqueName: \"kubernetes.io/projected/ab65a1ba-c07c-4930-ac5b-e80eaadfe184-kube-api-access-vjv58\") pod \"payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq\" (UID: \"ab65a1ba-c07c-4930-ac5b-e80eaadfe184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"
Nov 02 12:10:35.614589 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:35.614550    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab65a1ba-c07c-4930-ac5b-e80eaadfe184-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq\" (UID: \"ab65a1ba-c07c-4930-ac5b-e80eaadfe184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"
Nov 02 12:10:35.715445 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:35.715367    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjv58\" (UniqueName: \"kubernetes.io/projected/ab65a1ba-c07c-4930-ac5b-e80eaadfe184-kube-api-access-vjv58\") pod \"payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq\" (UID: \"ab65a1ba-c07c-4930-ac5b-e80eaadfe184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"
Nov 02 12:10:35.715585 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:35.715465    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab65a1ba-c07c-4930-ac5b-e80eaadfe184-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq\" (UID: \"ab65a1ba-c07c-4930-ac5b-e80eaadfe184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"
Nov 02 12:10:35.716149 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:35.716131    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab65a1ba-c07c-4930-ac5b-e80eaadfe184-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq\" (UID: \"ab65a1ba-c07c-4930-ac5b-e80eaadfe184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"
Nov 02 12:10:35.777429 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:35.777401    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjv58\" (UniqueName: \"kubernetes.io/projected/ab65a1ba-c07c-4930-ac5b-e80eaadfe184-kube-api-access-vjv58\") pod \"payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq\" (UID: \"ab65a1ba-c07c-4930-ac5b-e80eaadfe184\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"
Nov 02 12:10:36.065273 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:36.065243    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"
Nov 02 12:10:36.436773 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:36.436746    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"]
Nov 02 12:10:36.437960 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:10:36.437918    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab65a1ba_c07c_4930_ac5b_e80eaadfe184.slice/crio-d12153f99a6981edc72e2465218c6e1fbf6c894e3bbd2e131224a2d00c064595 WatchSource:0}: Error finding container d12153f99a6981edc72e2465218c6e1fbf6c894e3bbd2e131224a2d00c064595: Status 404 returned error can't find the container with id d12153f99a6981edc72e2465218c6e1fbf6c894e3bbd2e131224a2d00c064595
Nov 02 12:10:36.699669 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:36.699592    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq" event={"ID":"ab65a1ba-c07c-4930-ac5b-e80eaadfe184","Type":"ContainerStarted","Data":"0f84511edd9f7198fa05d8bc997d46b8b72d88df495481fc2a25bf0ddf4687a1"}
Nov 02 12:10:36.699817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:36.699668    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq" event={"ID":"ab65a1ba-c07c-4930-ac5b-e80eaadfe184","Type":"ContainerStarted","Data":"d12153f99a6981edc72e2465218c6e1fbf6c894e3bbd2e131224a2d00c064595"}
Nov 02 12:10:36.699817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:36.699719    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"
Nov 02 12:10:36.821362 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:36.821307    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq" podStartSLOduration=1.821290723 podStartE2EDuration="1.821290723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:10:36.818491149 +0000 UTC m=+865.382199564" watchObservedRunningTime="2026-11-02 12:10:36.821290723 +0000 UTC m=+865.384999171"
Nov 02 12:10:47.708125 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:47.708092    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"
Nov 02 12:10:53.478038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.478005    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"]
Nov 02 12:10:53.482149 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.482133    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.539998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.536087    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7bd72e-openshift-default-dockercfg-tsgw4\""
Nov 02 12:10:53.643063 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.608119    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-data\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.643548 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.643517    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-podinfo\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.643736 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.643648    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-workload-certs\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.643887 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.643869    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-token\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.644016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.644002    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-envoy\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.644160 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.644147    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-workload-socket\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.644283 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.644269    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-credential-socket\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.644379 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.644365    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istiod-ca-cert\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.644475 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.644463    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gplw2\" (UniqueName: \"kubernetes.io/projected/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-kube-api-access-gplw2\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.674871 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.674836    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"]
Nov 02 12:10:53.745819 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.745750    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-data\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.745963 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.745828    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-podinfo\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.745963 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.745847    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-workload-certs\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.745963 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.745870    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-token\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.745963 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.745893    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-envoy\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.746188 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.746024    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-workload-socket\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.746188 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.746064    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-credential-socket\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.746188 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.746158    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istiod-ca-cert\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.746363 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.746204    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplw2\" (UniqueName: \"kubernetes.io/projected/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-kube-api-access-gplw2\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.746363 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.746209    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-data\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.746363 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.746315    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-workload-certs\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.746533 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.746390    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-credential-socket\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.746715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.746693    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-workload-socket\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.747024 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.747002    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istiod-ca-cert\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.748156 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.748138    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-envoy\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.748573 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.748546    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-podinfo\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.808197 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.808167    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-token\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:53.808391 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:53.808372    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gplw2\" (UniqueName: \"kubernetes.io/projected/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-kube-api-access-gplw2\") pod \"e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") " pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:54.097498 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:54.097417    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:54.524550 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:54.524512    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"]
Nov 02 12:10:54.560233 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:10:54.560204    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode127e191_c68b_4d5f_8c6c_d6b42598e2b6.slice/crio-5ba48c14181ff116b155596d79797253adb78d5e5ce1c57af48c88bf22efa11e WatchSource:0}: Error finding container 5ba48c14181ff116b155596d79797253adb78d5e5ce1c57af48c88bf22efa11e: Status 404 returned error can't find the container with id 5ba48c14181ff116b155596d79797253adb78d5e5ce1c57af48c88bf22efa11e
Nov 02 12:10:54.562414 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:54.562380    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:10:54.562522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:54.562459    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:10:54.562522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:54.562500    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:10:54.866511 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:54.866431    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd" event={"ID":"e127e191-c68b-4d5f-8c6c-d6b42598e2b6","Type":"ContainerStarted","Data":"ddb98e1aa1f928a20c72e4980aa69c9eb00050ecaa7c980237aba499d493ecbf"}
Nov 02 12:10:54.866511 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:54.866493    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd" event={"ID":"e127e191-c68b-4d5f-8c6c-d6b42598e2b6","Type":"ContainerStarted","Data":"5ba48c14181ff116b155596d79797253adb78d5e5ce1c57af48c88bf22efa11e"}
Nov 02 12:10:54.989471 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:54.989420    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd" podStartSLOduration=1.9894068919999999 podStartE2EDuration="1.989406892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:10:54.988523701 +0000 UTC m=+883.552232131" watchObservedRunningTime="2026-11-02 12:10:54.989406892 +0000 UTC m=+883.553115301"
Nov 02 12:10:55.103808 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:55.102555    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:55.124744 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:55.121701    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:55.873033 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:55.873001    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:55.873823 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:10:55.873805    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:10:58.291771 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:10:58.291739    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:11:02.766882 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:02.766849    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:11:11.651287 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:11.651256    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:11:11.653365 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:11.653344    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:11:11.668843 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:11.668820    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:11:11.670742 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:11.670722    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:11:12.863396 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:12.863360    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"]
Nov 02 12:11:12.868726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:12.868705    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:12.921925 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:12.921898    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b1703c-openshift-default-dockercfg-rb2rb\""
Nov 02 12:11:12.975927 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:12.975901    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"]
Nov 02 12:11:12.988398 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:12.988366    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda61e72-8ab1-4390-8295-83a8fc113578-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:12.988809 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:12.988774    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:12.989035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:12.989016    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda61e72-8ab1-4390-8295-83a8fc113578-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:12.989200 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:12.989182    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:12.989334 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:12.989317    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qctgd\" (UniqueName: \"kubernetes.io/projected/dda61e72-8ab1-4390-8295-83a8fc113578-kube-api-access-qctgd\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:12.989445 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:12.989431    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:12.989580 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:12.989566    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:12.989708 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:12.989693    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:12.989827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:12.989812    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda61e72-8ab1-4390-8295-83a8fc113578-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.091550 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.091521    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda61e72-8ab1-4390-8295-83a8fc113578-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.091733 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.091566    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.091733 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.091591    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda61e72-8ab1-4390-8295-83a8fc113578-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.091733 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.091631    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.091733 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.091672    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qctgd\" (UniqueName: \"kubernetes.io/projected/dda61e72-8ab1-4390-8295-83a8fc113578-kube-api-access-qctgd\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.091733 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.091699    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.092001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.091753    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.092001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.091804    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.092001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.091848    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda61e72-8ab1-4390-8295-83a8fc113578-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.092170 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.092000    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.092170 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.092047    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.092341 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.092323    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.092428 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.092408    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.092703 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.092666    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda61e72-8ab1-4390-8295-83a8fc113578-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.094699 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.094675    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.095156 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.095121    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda61e72-8ab1-4390-8295-83a8fc113578-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.153520 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.153452    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda61e72-8ab1-4390-8295-83a8fc113578-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.153640 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.153577    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qctgd\" (UniqueName: \"kubernetes.io/projected/dda61e72-8ab1-4390-8295-83a8fc113578-kube-api-access-qctgd\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.182092 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.182067    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:13.619033 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.618992    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"]
Nov 02 12:11:13.654700 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:11:13.654652    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda61e72_8ab1_4390_8295_83a8fc113578.slice/crio-91fe58c626101525e77640b04fb89d7bf262f67b6c94ade3f9f08e11b4cd7bcd WatchSource:0}: Error finding container 91fe58c626101525e77640b04fb89d7bf262f67b6c94ade3f9f08e11b4cd7bcd: Status 404 returned error can't find the container with id 91fe58c626101525e77640b04fb89d7bf262f67b6c94ade3f9f08e11b4cd7bcd
Nov 02 12:11:13.659363 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.658908    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:11:13.667558 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.659100    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:11:13.667558 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:13.667189    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:11:14.013039 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:14.009379    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p" event={"ID":"dda61e72-8ab1-4390-8295-83a8fc113578","Type":"ContainerStarted","Data":"d5aa3a11acdd3c1d9a91f07eb8c1c2f3d2ea4c1a9fe5333a29b2eaac8d901ead"}
Nov 02 12:11:14.013039 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:14.009520    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p" event={"ID":"dda61e72-8ab1-4390-8295-83a8fc113578","Type":"ContainerStarted","Data":"91fe58c626101525e77640b04fb89d7bf262f67b6c94ade3f9f08e11b4cd7bcd"}
Nov 02 12:11:14.153016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:14.152914    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p" podStartSLOduration=2.15289688 podStartE2EDuration="2.15289688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:11:14.152079537 +0000 UTC m=+902.715787947" watchObservedRunningTime="2026-11-02 12:11:14.15289688 +0000 UTC m=+902.716605290"
Nov 02 12:11:14.186000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:14.185362    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:14.200007 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:14.194249    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:11:14.200007 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:14.194306    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p" podUID="dda61e72-8ab1-4390-8295-83a8fc113578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 12:11:15.186444 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:15.186403    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:11:15.187197 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:15.187164    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p" podUID="dda61e72-8ab1-4390-8295-83a8fc113578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 12:11:16.185999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:16.185357    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:11:16.185999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:16.185435    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p" podUID="dda61e72-8ab1-4390-8295-83a8fc113578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 12:11:17.186380 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:17.186343    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:11:17.186996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:17.186941    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p" podUID="dda61e72-8ab1-4390-8295-83a8fc113578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 12:11:18.195826 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:18.195343    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:11:18.195826 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:18.195406    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p" podUID="dda61e72-8ab1-4390-8295-83a8fc113578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 12:11:19.184402 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:19.184365    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:11:19.184606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:19.184464    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p" podUID="dda61e72-8ab1-4390-8295-83a8fc113578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 12:11:20.183305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:20.183271    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:11:20.183725 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:20.183335    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p" podUID="dda61e72-8ab1-4390-8295-83a8fc113578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 02 12:11:21.187269 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:21.187239    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:21.187746 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:21.187551    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:21.188114 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:21.188094    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:11:23.062437 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:23.062407    4464 generic.go:356] "Generic (PLEG): container finished" podID="9d39d963-d570-44cc-96fd-a4efee1e45fc" containerID="b5870258acae6a70a0cdf5fc7482cf5c13d061212278229a91117c59090dbf4f" exitCode=0
Nov 02 12:11:23.062873 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:23.062482    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d" event={"ID":"9d39d963-d570-44cc-96fd-a4efee1e45fc","Type":"ContainerDied","Data":"b5870258acae6a70a0cdf5fc7482cf5c13d061212278229a91117c59090dbf4f"}
Nov 02 12:11:23.062873 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:23.062858    4464 scope.go:122] "RemoveContainer" containerID="b5870258acae6a70a0cdf5fc7482cf5c13d061212278229a91117c59090dbf4f"
Nov 02 12:11:23.064025 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:23.064002    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:11:23.064089 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:23.064063    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:11:23.064125 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:23.064092    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:11:24.088755 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:24.088715    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d" event={"ID":"9d39d963-d570-44cc-96fd-a4efee1e45fc","Type":"ContainerStarted","Data":"a41d4fb954400960308b10726dca8e523cc238e6c5170d13757177deda8f33dd"}
Nov 02 12:11:24.205494 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:24.205461    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:11:24.205723 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:24.205703    4464 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 12:11:24.205777 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:24.205746    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d" podUID="9d39d963-d570-44cc-96fd-a4efee1e45fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 12:11:25.205637 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:25.205604    4464 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 12:11:25.206073 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:25.205654    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d" podUID="9d39d963-d570-44cc-96fd-a4efee1e45fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 12:11:26.047625 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:26.047592    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:11:26.205305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:26.205274    4464 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 12:11:26.205479 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:26.205323    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d" podUID="9d39d963-d570-44cc-96fd-a4efee1e45fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 12:11:27.205536 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:27.205499    4464 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 12:11:27.205959 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:27.205552    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d" podUID="9d39d963-d570-44cc-96fd-a4efee1e45fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 12:11:28.205546 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:28.205510    4464 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 12:11:28.205944 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:28.205577    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d" podUID="9d39d963-d570-44cc-96fd-a4efee1e45fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 12:11:28.310892 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:11:28.310838    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:11:29.208941 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:29.208905    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:11:29.209350 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:29.209259    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:11:29.209915 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:29.209899    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:11:30.583843 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:30.583809    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"]
Nov 02 12:11:30.587947 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:30.587925    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"
Nov 02 12:11:30.645917 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:30.645882    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"]
Nov 02 12:11:30.661336 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:30.661307    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhm9z\" (UniqueName: \"kubernetes.io/projected/912ce582-7325-436e-bf7b-6d3b98034bb7-kube-api-access-vhm9z\") pod \"maas-api-revoke-sub-fc3a5d26-smbzj\" (UID: \"912ce582-7325-436e-bf7b-6d3b98034bb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"
Nov 02 12:11:30.661510 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:30.661355    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912ce582-7325-436e-bf7b-6d3b98034bb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc3a5d26-smbzj\" (UID: \"912ce582-7325-436e-bf7b-6d3b98034bb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"
Nov 02 12:11:30.661575 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:30.661522    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912ce582-7325-436e-bf7b-6d3b98034bb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc3a5d26-smbzj\" (UID: \"912ce582-7325-436e-bf7b-6d3b98034bb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"
Nov 02 12:11:30.693509 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:30.693455    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:11:30.693509 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:30.693474    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:11:30.693689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:30.693474    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:11:30.693689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:30.693510    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:11:30.762666 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:30.762637    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhm9z\" (UniqueName: \"kubernetes.io/projected/912ce582-7325-436e-bf7b-6d3b98034bb7-kube-api-access-vhm9z\") pod \"maas-api-revoke-sub-fc3a5d26-smbzj\" (UID: \"912ce582-7325-436e-bf7b-6d3b98034bb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"
Nov 02 12:11:30.762806 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:30.762675    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912ce582-7325-436e-bf7b-6d3b98034bb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc3a5d26-smbzj\" (UID: \"912ce582-7325-436e-bf7b-6d3b98034bb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"
Nov 02 12:11:30.762806 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:30.762722    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912ce582-7325-436e-bf7b-6d3b98034bb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc3a5d26-smbzj\" (UID: \"912ce582-7325-436e-bf7b-6d3b98034bb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"
Nov 02 12:11:30.763445 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:30.763424    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912ce582-7325-436e-bf7b-6d3b98034bb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc3a5d26-smbzj\" (UID: \"912ce582-7325-436e-bf7b-6d3b98034bb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"
Nov 02 12:11:30.764947 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:30.764930    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912ce582-7325-436e-bf7b-6d3b98034bb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc3a5d26-smbzj\" (UID: \"912ce582-7325-436e-bf7b-6d3b98034bb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"
Nov 02 12:11:30.825015 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:30.824962    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhm9z\" (UniqueName: \"kubernetes.io/projected/912ce582-7325-436e-bf7b-6d3b98034bb7-kube-api-access-vhm9z\") pod \"maas-api-revoke-sub-fc3a5d26-smbzj\" (UID: \"912ce582-7325-436e-bf7b-6d3b98034bb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"
Nov 02 12:11:30.900945 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:30.900874    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"
Nov 02 12:11:31.273487 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:31.273461    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"]
Nov 02 12:11:31.274751 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:11:31.274716    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912ce582_7325_436e_bf7b_6d3b98034bb7.slice/crio-36e673e520b42aaddef90dbcaa3124ef5c001785cf0a2bed5b11d5c092b94335 WatchSource:0}: Error finding container 36e673e520b42aaddef90dbcaa3124ef5c001785cf0a2bed5b11d5c092b94335: Status 404 returned error can't find the container with id 36e673e520b42aaddef90dbcaa3124ef5c001785cf0a2bed5b11d5c092b94335
Nov 02 12:11:32.136690 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:32.136652    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj" event={"ID":"912ce582-7325-436e-bf7b-6d3b98034bb7","Type":"ContainerStarted","Data":"2984ee6c41dc6c6a27157cc6888c7b66bbf65909bacdb817262ac699cbad9494"}
Nov 02 12:11:32.137071 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:32.136760    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj" event={"ID":"912ce582-7325-436e-bf7b-6d3b98034bb7","Type":"ContainerStarted","Data":"36e673e520b42aaddef90dbcaa3124ef5c001785cf0a2bed5b11d5c092b94335"}
Nov 02 12:11:32.254942 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:32.254893    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj" podStartSLOduration=2.254879676 podStartE2EDuration="2.254879676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:11:32.254122787 +0000 UTC m=+920.817831198" watchObservedRunningTime="2026-11-02 12:11:32.254879676 +0000 UTC m=+920.818588121"
Nov 02 12:11:33.143310 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:33.143274    4464 generic.go:356] "Generic (PLEG): container finished" podID="912ce582-7325-436e-bf7b-6d3b98034bb7" containerID="2984ee6c41dc6c6a27157cc6888c7b66bbf65909bacdb817262ac699cbad9494" exitCode=0
Nov 02 12:11:33.143812 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:33.143352    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj" event={"ID":"912ce582-7325-436e-bf7b-6d3b98034bb7","Type":"ContainerDied","Data":"2984ee6c41dc6c6a27157cc6888c7b66bbf65909bacdb817262ac699cbad9494"}
Nov 02 12:11:34.294135 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:34.294111    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"
Nov 02 12:11:34.442923 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:34.442846    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/912ce582-7325-436e-bf7b-6d3b98034bb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912ce582-7325-436e-bf7b-6d3b98034bb7-maas-api-xks-ca\") pod \"912ce582-7325-436e-bf7b-6d3b98034bb7\" (UID: \"912ce582-7325-436e-bf7b-6d3b98034bb7\") "
Nov 02 12:11:34.442923 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:34.442898    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/912ce582-7325-436e-bf7b-6d3b98034bb7-kube-api-access-vhm9z\" (UniqueName: \"kubernetes.io/projected/912ce582-7325-436e-bf7b-6d3b98034bb7-kube-api-access-vhm9z\") pod \"912ce582-7325-436e-bf7b-6d3b98034bb7\" (UID: \"912ce582-7325-436e-bf7b-6d3b98034bb7\") "
Nov 02 12:11:34.443114 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:34.443042    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/912ce582-7325-436e-bf7b-6d3b98034bb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912ce582-7325-436e-bf7b-6d3b98034bb7-maas-api-openshift-ca\") pod \"912ce582-7325-436e-bf7b-6d3b98034bb7\" (UID: \"912ce582-7325-436e-bf7b-6d3b98034bb7\") "
Nov 02 12:11:34.443467 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:34.443427    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912ce582-7325-436e-bf7b-6d3b98034bb7-maas-api-openshift-ca" pod "912ce582-7325-436e-bf7b-6d3b98034bb7" (UID: "912ce582-7325-436e-bf7b-6d3b98034bb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:11:34.445231 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:34.445197    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912ce582-7325-436e-bf7b-6d3b98034bb7-kube-api-access-vhm9z" pod "912ce582-7325-436e-bf7b-6d3b98034bb7" (UID: "912ce582-7325-436e-bf7b-6d3b98034bb7"). InnerVolumeSpecName "kube-api-access-vhm9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:11:34.445231 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:34.445202    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/912ce582-7325-436e-bf7b-6d3b98034bb7-maas-api-xks-ca" pod "912ce582-7325-436e-bf7b-6d3b98034bb7" (UID: "912ce582-7325-436e-bf7b-6d3b98034bb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:11:34.545075 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:34.545051    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912ce582-7325-436e-bf7b-6d3b98034bb7-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:11:34.545075 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:34.545075    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhm9z\" (UniqueName: \"kubernetes.io/projected/912ce582-7325-436e-bf7b-6d3b98034bb7-kube-api-access-vhm9z\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:11:34.545213 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:34.545086    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912ce582-7325-436e-bf7b-6d3b98034bb7-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:11:35.158179 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:35.158140    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj" event={"ID":"912ce582-7325-436e-bf7b-6d3b98034bb7","Type":"ContainerDied","Data":"36e673e520b42aaddef90dbcaa3124ef5c001785cf0a2bed5b11d5c092b94335"}
Nov 02 12:11:35.158333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:35.158182    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"
Nov 02 12:11:35.158333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:35.158205    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36e673e520b42aaddef90dbcaa3124ef5c001785cf0a2bed5b11d5c092b94335"
Nov 02 12:11:42.796527 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.796495    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"]
Nov 02 12:11:42.797125 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.797109    4464 cpu_manager.go:414] "containerMap: removing container" podUID="912ce582-7325-436e-bf7b-6d3b98034bb7" containerName="revoke-keys"
Nov 02 12:11:42.797182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.797126    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="912ce582-7325-436e-bf7b-6d3b98034bb7" containerName="revoke-keys"
Nov 02 12:11:42.797230 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.797221    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="912ce582-7325-436e-bf7b-6d3b98034bb7" containerName="revoke-keys"
Nov 02 12:11:42.800662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.800647    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.838823 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.838791    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.838960 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.838867    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.839043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.838970    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4xq5\" (UniqueName: \"kubernetes.io/projected/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-kube-api-access-d4xq5\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.839043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.839035    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.839125 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.839063    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.839125 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.839108    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.839201 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.839174    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.839239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.839204    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.839239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.839224    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.900916 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.900890    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"]
Nov 02 12:11:42.940263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.940231    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.940424 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.940273    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.940424 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.940384    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.940536 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.940425    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.940536 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.940488    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.940536 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.940535    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4xq5\" (UniqueName: \"kubernetes.io/projected/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-kube-api-access-d4xq5\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.940697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.940564    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.940697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.940580    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.940697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.940606    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.940898 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.940758    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.940898 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.940880    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.941026 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.940943    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.941098 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.941021    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.941402 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.941381    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.942548 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.942526    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:42.943088 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:42.943065    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:43.002951 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:43.002924    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4xq5\" (UniqueName: \"kubernetes.io/projected/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-kube-api-access-d4xq5\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:43.003910 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:43.003893    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:43.113728 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:43.113657    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:43.541891 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:43.541860    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"]
Nov 02 12:11:43.542459 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:11:43.542431    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb8c3c9_4f23_4f81_806a_4d89eab16c48.slice/crio-4a497081055e2fb520dae8a2bda6ab4aa2c246dc11ee07723e30839197c770ae WatchSource:0}: Error finding container 4a497081055e2fb520dae8a2bda6ab4aa2c246dc11ee07723e30839197c770ae: Status 404 returned error can't find the container with id 4a497081055e2fb520dae8a2bda6ab4aa2c246dc11ee07723e30839197c770ae
Nov 02 12:11:43.544729 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:43.544700    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:11:43.544784 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:43.544763    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:11:43.544823 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:43.544794    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:11:44.230046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:44.230009    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7" event={"ID":"dfb8c3c9-4f23-4f81-806a-4d89eab16c48","Type":"ContainerStarted","Data":"ce6c1613298834fd7d77d28dbe0031064629b693c74cfbbe2956fc4dd171acbd"}
Nov 02 12:11:44.230439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:44.230072    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7" event={"ID":"dfb8c3c9-4f23-4f81-806a-4d89eab16c48","Type":"ContainerStarted","Data":"4a497081055e2fb520dae8a2bda6ab4aa2c246dc11ee07723e30839197c770ae"}
Nov 02 12:11:44.405566 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:44.405495    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7" podStartSLOduration=2.40548053 podStartE2EDuration="2.40548053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:11:44.402859062 +0000 UTC m=+932.966567471" watchObservedRunningTime="2026-11-02 12:11:44.40548053 +0000 UTC m=+932.969188939"
Nov 02 12:11:45.113928 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:45.113895    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:45.115571 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:45.115550    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 12:11:45.115654 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:45.115593    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7" podUID="dfb8c3c9-4f23-4f81-806a-4d89eab16c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 12:11:46.114161 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:46.114127    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 12:11:46.114595 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:46.114192    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7" podUID="dfb8c3c9-4f23-4f81-806a-4d89eab16c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 12:11:46.249257 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:46.248436    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:11:47.117465 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:47.117430    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 12:11:47.118097 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:47.118066    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7" podUID="dfb8c3c9-4f23-4f81-806a-4d89eab16c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 12:11:48.116007 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:48.115405    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 12:11:48.116007 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:48.115473    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7" podUID="dfb8c3c9-4f23-4f81-806a-4d89eab16c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 12:11:49.114302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:49.114212    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 12:11:49.114302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:49.114275    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7" podUID="dfb8c3c9-4f23-4f81-806a-4d89eab16c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 12:11:50.114437 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.114394    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 02 12:11:50.115024 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.114472    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7" podUID="dfb8c3c9-4f23-4f81-806a-4d89eab16c48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 02 12:11:50.248499 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.248468    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"]
Nov 02 12:11:50.253577 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.253557    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"
Nov 02 12:11:50.307324 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.307281    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:11:50.310646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.310616    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"]
Nov 02 12:11:50.357176 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.357141    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:11:50.357176 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.357141    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:11:50.357358 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.357156    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:11:50.443257 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.443168    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d780d2f-1755-4056-a811-72279983a394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cf582f8-72g4l\" (UID: \"6d780d2f-1755-4056-a811-72279983a394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"
Nov 02 12:11:50.443401 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.443370    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l96xs\" (UniqueName: \"kubernetes.io/projected/6d780d2f-1755-4056-a811-72279983a394-kube-api-access-l96xs\") pod \"maas-api-revoke-sub-0cf582f8-72g4l\" (UID: \"6d780d2f-1755-4056-a811-72279983a394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"
Nov 02 12:11:50.443461 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.443405    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d780d2f-1755-4056-a811-72279983a394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cf582f8-72g4l\" (UID: \"6d780d2f-1755-4056-a811-72279983a394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"
Nov 02 12:11:50.545029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.544960    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l96xs\" (UniqueName: \"kubernetes.io/projected/6d780d2f-1755-4056-a811-72279983a394-kube-api-access-l96xs\") pod \"maas-api-revoke-sub-0cf582f8-72g4l\" (UID: \"6d780d2f-1755-4056-a811-72279983a394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"
Nov 02 12:11:50.545226 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.545054    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d780d2f-1755-4056-a811-72279983a394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cf582f8-72g4l\" (UID: \"6d780d2f-1755-4056-a811-72279983a394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"
Nov 02 12:11:50.545226 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.545132    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d780d2f-1755-4056-a811-72279983a394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cf582f8-72g4l\" (UID: \"6d780d2f-1755-4056-a811-72279983a394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"
Nov 02 12:11:50.545874 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.545849    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d780d2f-1755-4056-a811-72279983a394-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cf582f8-72g4l\" (UID: \"6d780d2f-1755-4056-a811-72279983a394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"
Nov 02 12:11:50.547698 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.547670    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d780d2f-1755-4056-a811-72279983a394-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cf582f8-72g4l\" (UID: \"6d780d2f-1755-4056-a811-72279983a394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"
Nov 02 12:11:50.608679 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.608644    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l96xs\" (UniqueName: \"kubernetes.io/projected/6d780d2f-1755-4056-a811-72279983a394-kube-api-access-l96xs\") pod \"maas-api-revoke-sub-0cf582f8-72g4l\" (UID: \"6d780d2f-1755-4056-a811-72279983a394\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"
Nov 02 12:11:50.895851 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:50.895804    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"
Nov 02 12:11:51.293638 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:51.293608    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"]
Nov 02 12:11:51.296398 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:11:51.296366    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d780d2f_1755_4056_a811_72279983a394.slice/crio-e3ade6d3ca76dc1cc7971f866e91fdc8a21e4390e7897c2f119125420112c132 WatchSource:0}: Error finding container e3ade6d3ca76dc1cc7971f866e91fdc8a21e4390e7897c2f119125420112c132: Status 404 returned error can't find the container with id e3ade6d3ca76dc1cc7971f866e91fdc8a21e4390e7897c2f119125420112c132
Nov 02 12:11:51.324809 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:51.324786    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:51.325208 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:51.325186    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:51.325915 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:51.325895    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:11:51.345031 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:51.345004    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l" event={"ID":"6d780d2f-1755-4056-a811-72279983a394","Type":"ContainerStarted","Data":"e3ade6d3ca76dc1cc7971f866e91fdc8a21e4390e7897c2f119125420112c132"}
Nov 02 12:11:52.353035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:52.353002    4464 generic.go:356] "Generic (PLEG): container finished" podID="6d780d2f-1755-4056-a811-72279983a394" containerID="9cc6de1e88d1935edf60e35161df8aab8936113c54b603369d22a981d21e801f" exitCode=0
Nov 02 12:11:52.353385 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:52.353082    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l" event={"ID":"6d780d2f-1755-4056-a811-72279983a394","Type":"ContainerDied","Data":"9cc6de1e88d1935edf60e35161df8aab8936113c54b603369d22a981d21e801f"}
Nov 02 12:11:53.518742 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:53.518710    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"
Nov 02 12:11:53.602948 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:53.602915    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d780d2f-1755-4056-a811-72279983a394-kube-api-access-l96xs\" (UniqueName: \"kubernetes.io/projected/6d780d2f-1755-4056-a811-72279983a394-kube-api-access-l96xs\") pod \"6d780d2f-1755-4056-a811-72279983a394\" (UID: \"6d780d2f-1755-4056-a811-72279983a394\") "
Nov 02 12:11:53.603154 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:53.603111    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d780d2f-1755-4056-a811-72279983a394-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d780d2f-1755-4056-a811-72279983a394-maas-api-openshift-ca\") pod \"6d780d2f-1755-4056-a811-72279983a394\" (UID: \"6d780d2f-1755-4056-a811-72279983a394\") "
Nov 02 12:11:53.603208 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:53.603177    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d780d2f-1755-4056-a811-72279983a394-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d780d2f-1755-4056-a811-72279983a394-maas-api-xks-ca\") pod \"6d780d2f-1755-4056-a811-72279983a394\" (UID: \"6d780d2f-1755-4056-a811-72279983a394\") "
Nov 02 12:11:53.603807 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:53.603523    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d780d2f-1755-4056-a811-72279983a394-maas-api-openshift-ca" pod "6d780d2f-1755-4056-a811-72279983a394" (UID: "6d780d2f-1755-4056-a811-72279983a394"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:11:53.605466 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:53.605436    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d780d2f-1755-4056-a811-72279983a394-kube-api-access-l96xs" pod "6d780d2f-1755-4056-a811-72279983a394" (UID: "6d780d2f-1755-4056-a811-72279983a394"). InnerVolumeSpecName "kube-api-access-l96xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:11:53.605697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:53.605662    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d780d2f-1755-4056-a811-72279983a394-maas-api-xks-ca" pod "6d780d2f-1755-4056-a811-72279983a394" (UID: "6d780d2f-1755-4056-a811-72279983a394"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:11:53.704947 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:53.704906    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d780d2f-1755-4056-a811-72279983a394-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:11:53.704947 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:53.704943    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d780d2f-1755-4056-a811-72279983a394-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:11:53.704947 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:53.704954    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l96xs\" (UniqueName: \"kubernetes.io/projected/6d780d2f-1755-4056-a811-72279983a394-kube-api-access-l96xs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:11:54.371490 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:54.371238    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l" event={"ID":"6d780d2f-1755-4056-a811-72279983a394","Type":"ContainerDied","Data":"e3ade6d3ca76dc1cc7971f866e91fdc8a21e4390e7897c2f119125420112c132"}
Nov 02 12:11:54.371490 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:54.371345    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3ade6d3ca76dc1cc7971f866e91fdc8a21e4390e7897c2f119125420112c132"
Nov 02 12:11:54.371490 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:54.371422    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"
Nov 02 12:11:55.874807 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:55.874733    4464 patch_prober.go:29] interesting pod/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 12:11:55.874807 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:55.874782    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd" podUID="e127e191-c68b-4d5f-8c6c-d6b42598e2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 12:11:56.385455 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:56.385423    4464 generic.go:356] "Generic (PLEG): container finished" podID="e127e191-c68b-4d5f-8c6c-d6b42598e2b6" containerID="ddb98e1aa1f928a20c72e4980aa69c9eb00050ecaa7c980237aba499d493ecbf" exitCode=0
Nov 02 12:11:56.385627 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:56.385485    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd" event={"ID":"e127e191-c68b-4d5f-8c6c-d6b42598e2b6","Type":"ContainerDied","Data":"ddb98e1aa1f928a20c72e4980aa69c9eb00050ecaa7c980237aba499d493ecbf"}
Nov 02 12:11:56.385825 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:56.385812    4464 scope.go:122] "RemoveContainer" containerID="ddb98e1aa1f928a20c72e4980aa69c9eb00050ecaa7c980237aba499d493ecbf"
Nov 02 12:11:56.387079 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:56.387052    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:11:56.387150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:56.387118    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:11:56.387185 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:56.387150    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:11:57.394284 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:57.394251    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd" event={"ID":"e127e191-c68b-4d5f-8c6c-d6b42598e2b6","Type":"ContainerStarted","Data":"fc1515a341b1ff33b56ebb6021e16046fc87894a2bca733765ca6d0d0c167721"}
Nov 02 12:11:58.098228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.098185    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:11:58.098428 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.098372    4464 patch_prober.go:29] interesting pod/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 12:11:58.098505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.098422    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd" podUID="e127e191-c68b-4d5f-8c6c-d6b42598e2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 12:11:58.242225 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.242192    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"]
Nov 02 12:11:58.242899 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.242878    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6d780d2f-1755-4056-a811-72279983a394" containerName="revoke-keys"
Nov 02 12:11:58.242899 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.242899    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d780d2f-1755-4056-a811-72279983a394" containerName="revoke-keys"
Nov 02 12:11:58.243104 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.243060    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d780d2f-1755-4056-a811-72279983a394" containerName="revoke-keys"
Nov 02 12:11:58.245488 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.245467    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.315685 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:11:58.315646    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:11:58.341203 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.341157    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"]
Nov 02 12:11:58.354654 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.354569    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.354799 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.354654    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052217bc-c40c-4150-8769-32cc113feab6-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.354799 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.354691    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt6kg\" (UniqueName: \"kubernetes.io/projected/052217bc-c40c-4150-8769-32cc113feab6-kube-api-access-qt6kg\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.354933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.354806    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.354933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.354859    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.354933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.354906    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052217bc-c40c-4150-8769-32cc113feab6-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.355179 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.355030    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.355179 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.355048    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052217bc-c40c-4150-8769-32cc113feab6-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.355179 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.355072    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.456137 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.456105    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.456622 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.456171    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052217bc-c40c-4150-8769-32cc113feab6-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.456622 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.456195    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt6kg\" (UniqueName: \"kubernetes.io/projected/052217bc-c40c-4150-8769-32cc113feab6-kube-api-access-qt6kg\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.456622 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.456240    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.456622 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.456280    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.456622 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.456341    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052217bc-c40c-4150-8769-32cc113feab6-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.456622 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.456490    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.456622 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.456523    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052217bc-c40c-4150-8769-32cc113feab6-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.456622 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.456551    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.456622 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.456576    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.457004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.456696    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.457004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.456854    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.457004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.456902    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.457151 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.457090    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052217bc-c40c-4150-8769-32cc113feab6-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.458724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.458701    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.458892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.458876    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052217bc-c40c-4150-8769-32cc113feab6-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.518018 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.517958    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052217bc-c40c-4150-8769-32cc113feab6-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.518161 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.517965    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt6kg\" (UniqueName: \"kubernetes.io/projected/052217bc-c40c-4150-8769-32cc113feab6-kube-api-access-qt6kg\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:58.557371 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:58.557343    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:11:59.016756 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:59.016726    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"]
Nov 02 12:11:59.017441 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:11:59.017409    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052217bc_c40c_4150_8769_32cc113feab6.slice/crio-55c9a6a868d1b46eb72278d4ea46b45309dea5122d1149a25a2712cf2537b8e9 WatchSource:0}: Error finding container 55c9a6a868d1b46eb72278d4ea46b45309dea5122d1149a25a2712cf2537b8e9: Status 404 returned error can't find the container with id 55c9a6a868d1b46eb72278d4ea46b45309dea5122d1149a25a2712cf2537b8e9
Nov 02 12:11:59.019739 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:59.019708    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:11:59.019814 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:59.019797    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:11:59.019856 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:59.019826    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:11:59.098655 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:59.098629    4464 patch_prober.go:29] interesting pod/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 12:11:59.098781 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:59.098673    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd" podUID="e127e191-c68b-4d5f-8c6c-d6b42598e2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 12:11:59.407319 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:59.407228    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8" event={"ID":"052217bc-c40c-4150-8769-32cc113feab6","Type":"ContainerStarted","Data":"f2f2f9eaabd06e99776bbb21ad5277b6e9fb130f713939378dd387a946f5a7c9"}
Nov 02 12:11:59.407319 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:59.407301    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8" event={"ID":"052217bc-c40c-4150-8769-32cc113feab6","Type":"ContainerStarted","Data":"55c9a6a868d1b46eb72278d4ea46b45309dea5122d1149a25a2712cf2537b8e9"}
Nov 02 12:11:59.535286 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:59.535235    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8" podStartSLOduration=1.5352208109999999 podStartE2EDuration="1.535220811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:11:59.534079389 +0000 UTC m=+948.097787801" watchObservedRunningTime="2026-11-02 12:11:59.535220811 +0000 UTC m=+948.098929221"
Nov 02 12:11:59.557897 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:11:59.557872    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:12:00.098430 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:00.098392    4464 patch_prober.go:29] interesting pod/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 12:12:00.098756 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:00.098726    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd" podUID="e127e191-c68b-4d5f-8c6c-d6b42598e2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 12:12:00.314295 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:00.314257    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:12:00.572004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:00.571942    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 12:12:00.572837 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:00.572536    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8" podUID="052217bc-c40c-4150-8769-32cc113feab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 12:12:01.103000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:01.099444    4464 patch_prober.go:29] interesting pod/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 12:12:01.103000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:01.099531    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd" podUID="e127e191-c68b-4d5f-8c6c-d6b42598e2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 12:12:01.557746 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:01.557705    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 12:12:01.557918 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:01.557768    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8" podUID="052217bc-c40c-4150-8769-32cc113feab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 12:12:02.098405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:02.098363    4464 patch_prober.go:29] interesting pod/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 02 12:12:02.099154 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:02.099122    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd" podUID="e127e191-c68b-4d5f-8c6c-d6b42598e2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 02 12:12:02.558436 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:02.558373    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 12:12:02.558628 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:02.558475    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8" podUID="052217bc-c40c-4150-8769-32cc113feab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 12:12:03.102376 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:03.102344    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:12:03.102757 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:03.102601    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:12:03.103341 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:03.103326    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:12:03.557695 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:03.557667    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 12:12:03.557872 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:03.557719    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8" podUID="052217bc-c40c-4150-8769-32cc113feab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 12:12:04.558543 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:04.558507    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 12:12:04.558947 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:04.558566    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8" podUID="052217bc-c40c-4150-8769-32cc113feab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 12:12:05.558470 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:05.558438    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 02 12:12:05.558648 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:05.558492    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8" podUID="052217bc-c40c-4150-8769-32cc113feab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 02 12:12:07.220899 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:07.220860    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:12:07.221633 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:07.221605    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:12:07.222809 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:07.222790    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:12:08.396521 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:08.396487    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"]
Nov 02 12:12:08.400622 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:08.400605    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"
Nov 02 12:12:08.460388 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:08.460362    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"]
Nov 02 12:12:08.507542 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:08.507520    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:12:08.507542 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:08.507540    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:12:08.507722 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:08.507520    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:12:08.509588 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:08.509575    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:12:08.533967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:08.533937    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c8dc80-9a57-4624-be9b-8a73c1141dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-040dbd72-rwlsd\" (UID: \"47c8dc80-9a57-4624-be9b-8a73c1141dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"
Nov 02 12:12:08.534110 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:08.534093    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqgpw\" (UniqueName: \"kubernetes.io/projected/47c8dc80-9a57-4624-be9b-8a73c1141dff-kube-api-access-jqgpw\") pod \"maas-api-revoke-sub-040dbd72-rwlsd\" (UID: \"47c8dc80-9a57-4624-be9b-8a73c1141dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"
Nov 02 12:12:08.534215 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:08.534201    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c8dc80-9a57-4624-be9b-8a73c1141dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-040dbd72-rwlsd\" (UID: \"47c8dc80-9a57-4624-be9b-8a73c1141dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"
Nov 02 12:12:08.635045 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:08.635016    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c8dc80-9a57-4624-be9b-8a73c1141dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-040dbd72-rwlsd\" (UID: \"47c8dc80-9a57-4624-be9b-8a73c1141dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"
Nov 02 12:12:08.635177 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:08.635097    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqgpw\" (UniqueName: \"kubernetes.io/projected/47c8dc80-9a57-4624-be9b-8a73c1141dff-kube-api-access-jqgpw\") pod \"maas-api-revoke-sub-040dbd72-rwlsd\" (UID: \"47c8dc80-9a57-4624-be9b-8a73c1141dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"
Nov 02 12:12:08.635235 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:08.635195    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c8dc80-9a57-4624-be9b-8a73c1141dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-040dbd72-rwlsd\" (UID: \"47c8dc80-9a57-4624-be9b-8a73c1141dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"
Nov 02 12:12:08.635715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:08.635694    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c8dc80-9a57-4624-be9b-8a73c1141dff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-040dbd72-rwlsd\" (UID: \"47c8dc80-9a57-4624-be9b-8a73c1141dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"
Nov 02 12:12:08.637461 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:08.637439    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c8dc80-9a57-4624-be9b-8a73c1141dff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-040dbd72-rwlsd\" (UID: \"47c8dc80-9a57-4624-be9b-8a73c1141dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"
Nov 02 12:12:08.697165 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:08.697116    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqgpw\" (UniqueName: \"kubernetes.io/projected/47c8dc80-9a57-4624-be9b-8a73c1141dff-kube-api-access-jqgpw\") pod \"maas-api-revoke-sub-040dbd72-rwlsd\" (UID: \"47c8dc80-9a57-4624-be9b-8a73c1141dff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"
Nov 02 12:12:08.714074 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:08.714056    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"
Nov 02 12:12:09.088415 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:09.088383    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"]
Nov 02 12:12:09.089460 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:12:09.089434    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c8dc80_9a57_4624_be9b_8a73c1141dff.slice/crio-3601726abef5895d4ad0200737a7ee49a0508143bcac1a62181c2d717758c5a9 WatchSource:0}: Error finding container 3601726abef5895d4ad0200737a7ee49a0508143bcac1a62181c2d717758c5a9: Status 404 returned error can't find the container with id 3601726abef5895d4ad0200737a7ee49a0508143bcac1a62181c2d717758c5a9
Nov 02 12:12:09.488733 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:09.488694    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd" event={"ID":"47c8dc80-9a57-4624-be9b-8a73c1141dff","Type":"ContainerStarted","Data":"5aaefbc70d8f79b91842fb5307d82fd4210ccf47a8f25388c9382ebe0a57ab2a"}
Nov 02 12:12:09.489246 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:09.488761    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd" event={"ID":"47c8dc80-9a57-4624-be9b-8a73c1141dff","Type":"ContainerStarted","Data":"3601726abef5895d4ad0200737a7ee49a0508143bcac1a62181c2d717758c5a9"}
Nov 02 12:12:09.608907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:09.608857    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd" podStartSLOduration=1.6088430580000002 podStartE2EDuration="1.608843058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:12:09.606611981 +0000 UTC m=+958.170320391" watchObservedRunningTime="2026-11-02 12:12:09.608843058 +0000 UTC m=+958.172551466"
Nov 02 12:12:10.503194 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:10.503157    4464 generic.go:356] "Generic (PLEG): container finished" podID="47c8dc80-9a57-4624-be9b-8a73c1141dff" containerID="5aaefbc70d8f79b91842fb5307d82fd4210ccf47a8f25388c9382ebe0a57ab2a" exitCode=0
Nov 02 12:12:10.503922 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:10.503895    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd" event={"ID":"47c8dc80-9a57-4624-be9b-8a73c1141dff","Type":"ContainerDied","Data":"5aaefbc70d8f79b91842fb5307d82fd4210ccf47a8f25388c9382ebe0a57ab2a"}
Nov 02 12:12:11.676417 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:11.676392    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"
Nov 02 12:12:11.770097 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:11.770029    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c8dc80-9a57-4624-be9b-8a73c1141dff-kube-api-access-jqgpw\" (UniqueName: \"kubernetes.io/projected/47c8dc80-9a57-4624-be9b-8a73c1141dff-kube-api-access-jqgpw\") pod \"47c8dc80-9a57-4624-be9b-8a73c1141dff\" (UID: \"47c8dc80-9a57-4624-be9b-8a73c1141dff\") "
Nov 02 12:12:11.770227 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:11.770129    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47c8dc80-9a57-4624-be9b-8a73c1141dff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c8dc80-9a57-4624-be9b-8a73c1141dff-maas-api-xks-ca\") pod \"47c8dc80-9a57-4624-be9b-8a73c1141dff\" (UID: \"47c8dc80-9a57-4624-be9b-8a73c1141dff\") "
Nov 02 12:12:11.770227 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:11.770167    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47c8dc80-9a57-4624-be9b-8a73c1141dff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c8dc80-9a57-4624-be9b-8a73c1141dff-maas-api-openshift-ca\") pod \"47c8dc80-9a57-4624-be9b-8a73c1141dff\" (UID: \"47c8dc80-9a57-4624-be9b-8a73c1141dff\") "
Nov 02 12:12:11.770525 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:11.770497    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c8dc80-9a57-4624-be9b-8a73c1141dff-maas-api-openshift-ca" pod "47c8dc80-9a57-4624-be9b-8a73c1141dff" (UID: "47c8dc80-9a57-4624-be9b-8a73c1141dff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:12:11.770649 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:11.770633    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47c8dc80-9a57-4624-be9b-8a73c1141dff-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:12:11.772079 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:11.772055    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47c8dc80-9a57-4624-be9b-8a73c1141dff-maas-api-xks-ca" pod "47c8dc80-9a57-4624-be9b-8a73c1141dff" (UID: "47c8dc80-9a57-4624-be9b-8a73c1141dff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:12:11.772191 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:11.772161    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c8dc80-9a57-4624-be9b-8a73c1141dff-kube-api-access-jqgpw" pod "47c8dc80-9a57-4624-be9b-8a73c1141dff" (UID: "47c8dc80-9a57-4624-be9b-8a73c1141dff"). InnerVolumeSpecName "kube-api-access-jqgpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:12:11.872089 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:11.872057    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqgpw\" (UniqueName: \"kubernetes.io/projected/47c8dc80-9a57-4624-be9b-8a73c1141dff-kube-api-access-jqgpw\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:12:11.872089 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:11.872083    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47c8dc80-9a57-4624-be9b-8a73c1141dff-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:12:12.524491 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:12.524462    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"
Nov 02 12:12:12.524662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:12.524495    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd" event={"ID":"47c8dc80-9a57-4624-be9b-8a73c1141dff","Type":"ContainerDied","Data":"3601726abef5895d4ad0200737a7ee49a0508143bcac1a62181c2d717758c5a9"}
Nov 02 12:12:12.524662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:12.524543    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3601726abef5895d4ad0200737a7ee49a0508143bcac1a62181c2d717758c5a9"
Nov 02 12:12:13.070752 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.070715    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"]
Nov 02 12:12:13.071362 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.071346    4464 cpu_manager.go:414] "containerMap: removing container" podUID="47c8dc80-9a57-4624-be9b-8a73c1141dff" containerName="revoke-keys"
Nov 02 12:12:13.071406 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.071365    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c8dc80-9a57-4624-be9b-8a73c1141dff" containerName="revoke-keys"
Nov 02 12:12:13.071464 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.071454    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c8dc80-9a57-4624-be9b-8a73c1141dff" containerName="revoke-keys"
Nov 02 12:12:13.074785 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.074767    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.155896 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.155863    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"]
Nov 02 12:12:13.186534 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.186506    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.186688 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.186543    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.186688 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.186627    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.186763 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.186725    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.186824 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.186811    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.186859 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.186837    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.186890 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.186859    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-846gr\" (UniqueName: \"kubernetes.io/projected/e418db77-2841-458b-be0e-e9c69ee81ac1-kube-api-access-846gr\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.186944 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.186933    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.187003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.186965    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e418db77-2841-458b-be0e-e9c69ee81ac1-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.288133 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.288095    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.288310 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.288140    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e418db77-2841-458b-be0e-e9c69ee81ac1-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.288310 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.288185    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.288310 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.288212    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.288310 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.288264    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.288526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.288353    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.288526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.288439    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.288634 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.288530    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.288634 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.288576    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-846gr\" (UniqueName: \"kubernetes.io/projected/e418db77-2841-458b-be0e-e9c69ee81ac1-kube-api-access-846gr\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.288743 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.288661    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.288803 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.288736    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.288900 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.288877    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.289000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.288960    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e418db77-2841-458b-be0e-e9c69ee81ac1-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.289120 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.289103    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.290495 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.290460    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.290697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.290680    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.350008 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.349933    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.350168 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.350148    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-846gr\" (UniqueName: \"kubernetes.io/projected/e418db77-2841-458b-be0e-e9c69ee81ac1-kube-api-access-846gr\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.388696 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.388669    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:13.819771 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.816113    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"]
Nov 02 12:12:13.821287 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.821249    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:12:13.821381 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.821334    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:12:13.821381 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:13.821374    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:12:14.537175 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:14.537138    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7" event={"ID":"e418db77-2841-458b-be0e-e9c69ee81ac1","Type":"ContainerStarted","Data":"f9000f5976de400442688fe5b64c5b72fb62093139dacc38abeba370b129c778"}
Nov 02 12:12:14.537553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:14.537202    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7" event={"ID":"e418db77-2841-458b-be0e-e9c69ee81ac1","Type":"ContainerStarted","Data":"0e9c91b895d66a3ec62dcf0ad65c787bc1582cff8aa43045c8ecb7c0ee0376fd"}
Nov 02 12:12:14.673998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:14.659875    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7" podStartSLOduration=1.659857783 podStartE2EDuration="1.659857783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:12:14.659483842 +0000 UTC m=+963.223192252" watchObservedRunningTime="2026-11-02 12:12:14.659857783 +0000 UTC m=+963.223566195"
Nov 02 12:12:15.389539 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:15.389499    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:15.390969 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:15.390937    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:12:15.391083 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:15.391024    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7" podUID="e418db77-2841-458b-be0e-e9c69ee81ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 12:12:16.389005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:16.388962    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:12:16.389413 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:16.389024    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7" podUID="e418db77-2841-458b-be0e-e9c69ee81ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 12:12:17.389666 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:17.389636    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:12:17.390040 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:17.389682    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7" podUID="e418db77-2841-458b-be0e-e9c69ee81ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 12:12:18.389589 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:18.389555    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:12:18.389771 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:18.389605    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7" podUID="e418db77-2841-458b-be0e-e9c69ee81ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 12:12:19.390040 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:19.390003    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:12:19.390444 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:19.390061    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7" podUID="e418db77-2841-458b-be0e-e9c69ee81ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 12:12:20.390153 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:20.390113    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:12:20.390775 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:20.390741    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7" podUID="e418db77-2841-458b-be0e-e9c69ee81ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 12:12:21.612145 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:21.612097    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:21.613231 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:21.613206    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:21.613734 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:21.613713    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:12:25.288021 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:25.287956    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:12:28.292061 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:12:28.292028    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:12:34.546950 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.546914    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"]
Nov 02 12:12:34.551046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.551026    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"
Nov 02 12:12:34.630067 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.630028    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b8183e-899a-42ca-89b1-9a9db37b239a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v\" (UID: \"01b8183e-899a-42ca-89b1-9a9db37b239a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"
Nov 02 12:12:34.630503 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.630472    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78cpd\" (UniqueName: \"kubernetes.io/projected/01b8183e-899a-42ca-89b1-9a9db37b239a-kube-api-access-78cpd\") pod \"payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v\" (UID: \"01b8183e-899a-42ca-89b1-9a9db37b239a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"
Nov 02 12:12:34.672067 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.666407    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-036f88-dockercfg-gzzcd\""
Nov 02 12:12:34.674183 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.674138    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-036f88\""
Nov 02 12:12:34.733779 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.733723    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b8183e-899a-42ca-89b1-9a9db37b239a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v\" (UID: \"01b8183e-899a-42ca-89b1-9a9db37b239a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"
Nov 02 12:12:34.734179 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.734154    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78cpd\" (UniqueName: \"kubernetes.io/projected/01b8183e-899a-42ca-89b1-9a9db37b239a-kube-api-access-78cpd\") pod \"payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v\" (UID: \"01b8183e-899a-42ca-89b1-9a9db37b239a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"
Nov 02 12:12:34.735577 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.735552    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b8183e-899a-42ca-89b1-9a9db37b239a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v\" (UID: \"01b8183e-899a-42ca-89b1-9a9db37b239a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"
Nov 02 12:12:34.756939 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.756862    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"]
Nov 02 12:12:34.759849 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.759824    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"]
Nov 02 12:12:34.764285 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.764267    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"
Nov 02 12:12:34.765053 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.765029    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"]
Nov 02 12:12:34.797244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.797186    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78cpd\" (UniqueName: \"kubernetes.io/projected/01b8183e-899a-42ca-89b1-9a9db37b239a-kube-api-access-78cpd\") pod \"payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v\" (UID: \"01b8183e-899a-42ca-89b1-9a9db37b239a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"
Nov 02 12:12:34.835893 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.835868    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef3a987-e107-4351-9e85-cafeecf0a708-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-036f88-6588f64457-jfg29\" (UID: \"6ef3a987-e107-4351-9e85-cafeecf0a708\") " pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"
Nov 02 12:12:34.836018 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.835920    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf9f2\" (UniqueName: \"kubernetes.io/projected/6ef3a987-e107-4351-9e85-cafeecf0a708-kube-api-access-pf9f2\") pod \"payload-processing-e2e-inf-b-036f88-6588f64457-jfg29\" (UID: \"6ef3a987-e107-4351-9e85-cafeecf0a708\") " pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"
Nov 02 12:12:34.864743 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.864722    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"
Nov 02 12:12:34.936989 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.936939    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef3a987-e107-4351-9e85-cafeecf0a708-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-036f88-6588f64457-jfg29\" (UID: \"6ef3a987-e107-4351-9e85-cafeecf0a708\") " pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"
Nov 02 12:12:34.937128 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.937037    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf9f2\" (UniqueName: \"kubernetes.io/projected/6ef3a987-e107-4351-9e85-cafeecf0a708-kube-api-access-pf9f2\") pod \"payload-processing-e2e-inf-b-036f88-6588f64457-jfg29\" (UID: \"6ef3a987-e107-4351-9e85-cafeecf0a708\") " pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"
Nov 02 12:12:34.938390 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.938368    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef3a987-e107-4351-9e85-cafeecf0a708-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-036f88-6588f64457-jfg29\" (UID: \"6ef3a987-e107-4351-9e85-cafeecf0a708\") " pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"
Nov 02 12:12:34.999322 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:34.999292    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf9f2\" (UniqueName: \"kubernetes.io/projected/6ef3a987-e107-4351-9e85-cafeecf0a708-kube-api-access-pf9f2\") pod \"payload-processing-e2e-inf-b-036f88-6588f64457-jfg29\" (UID: \"6ef3a987-e107-4351-9e85-cafeecf0a708\") " pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"
Nov 02 12:12:35.080059 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:35.079958    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"
Nov 02 12:12:35.279464 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:35.279433    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"]
Nov 02 12:12:35.279711 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:12:35.279688    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b8183e_899a_42ca_89b1_9a9db37b239a.slice/crio-2d49a310af7b8f72ed7f4e6810b8dd682342209c1633a82ba1d34f8ba309593f WatchSource:0}: Error finding container 2d49a310af7b8f72ed7f4e6810b8dd682342209c1633a82ba1d34f8ba309593f: Status 404 returned error can't find the container with id 2d49a310af7b8f72ed7f4e6810b8dd682342209c1633a82ba1d34f8ba309593f
Nov 02 12:12:35.456745 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:12:35.456716    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef3a987_e107_4351_9e85_cafeecf0a708.slice/crio-1ac9cd7771bd1d45b16f5aabe327dceb63dc75a0334a10a673a8f895e8a136d2 WatchSource:0}: Error finding container 1ac9cd7771bd1d45b16f5aabe327dceb63dc75a0334a10a673a8f895e8a136d2: Status 404 returned error can't find the container with id 1ac9cd7771bd1d45b16f5aabe327dceb63dc75a0334a10a673a8f895e8a136d2
Nov 02 12:12:35.456874 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:35.456821    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"]
Nov 02 12:12:35.791031 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:35.790990    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v" event={"ID":"01b8183e-899a-42ca-89b1-9a9db37b239a","Type":"ContainerStarted","Data":"5265125648a116c355e29a8b6dd0975f34974a855a2acec498b8c2aaa8292a45"}
Nov 02 12:12:35.791494 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:35.791056    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v" event={"ID":"01b8183e-899a-42ca-89b1-9a9db37b239a","Type":"ContainerStarted","Data":"2d49a310af7b8f72ed7f4e6810b8dd682342209c1633a82ba1d34f8ba309593f"}
Nov 02 12:12:35.791494 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:35.791187    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"
Nov 02 12:12:35.792518 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:35.792492    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29" event={"ID":"6ef3a987-e107-4351-9e85-cafeecf0a708","Type":"ContainerStarted","Data":"b430a2037a7be05b52705adcb988d5fbc414c4f65a6fe50aaf5f08b610f46dbf"}
Nov 02 12:12:35.792622 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:35.792538    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29" event={"ID":"6ef3a987-e107-4351-9e85-cafeecf0a708","Type":"ContainerStarted","Data":"1ac9cd7771bd1d45b16f5aabe327dceb63dc75a0334a10a673a8f895e8a136d2"}
Nov 02 12:12:35.792622 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:35.792559    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"
Nov 02 12:12:35.911100 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:35.911054    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v" podStartSLOduration=1.911041457 podStartE2EDuration="1.911041457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:12:35.90825674 +0000 UTC m=+984.471965150" watchObservedRunningTime="2026-11-02 12:12:35.911041457 +0000 UTC m=+984.474749909"
Nov 02 12:12:36.063052 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:36.053423    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29" podStartSLOduration=2.053401044 podStartE2EDuration="2.053401044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:12:36.047951166 +0000 UTC m=+984.611659577" watchObservedRunningTime="2026-11-02 12:12:36.053401044 +0000 UTC m=+984.617109459"
Nov 02 12:12:39.874176 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:39.874137    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"]
Nov 02 12:12:39.879508 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:39.879485    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"
Nov 02 12:12:39.934004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:39.932817    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:12:39.936069 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:39.936023    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"]
Nov 02 12:12:39.982355 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:39.982328    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:12:39.983515 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:39.983496    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:12:40.020005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:40.015767    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnbmw\" (UniqueName: \"kubernetes.io/projected/6cb6bf60-cf44-4fff-ac7b-0995fced611e-kube-api-access-qnbmw\") pod \"maas-api-revoke-sub-481cb7b8-st2cn\" (UID: \"6cb6bf60-cf44-4fff-ac7b-0995fced611e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"
Nov 02 12:12:40.020005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:40.015909    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb6bf60-cf44-4fff-ac7b-0995fced611e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-481cb7b8-st2cn\" (UID: \"6cb6bf60-cf44-4fff-ac7b-0995fced611e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"
Nov 02 12:12:40.020005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:40.016034    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb6bf60-cf44-4fff-ac7b-0995fced611e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-481cb7b8-st2cn\" (UID: \"6cb6bf60-cf44-4fff-ac7b-0995fced611e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"
Nov 02 12:12:40.033225 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:40.033198    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:12:40.118005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:40.117882    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnbmw\" (UniqueName: \"kubernetes.io/projected/6cb6bf60-cf44-4fff-ac7b-0995fced611e-kube-api-access-qnbmw\") pod \"maas-api-revoke-sub-481cb7b8-st2cn\" (UID: \"6cb6bf60-cf44-4fff-ac7b-0995fced611e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"
Nov 02 12:12:40.121997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:40.118357    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb6bf60-cf44-4fff-ac7b-0995fced611e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-481cb7b8-st2cn\" (UID: \"6cb6bf60-cf44-4fff-ac7b-0995fced611e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"
Nov 02 12:12:40.121997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:40.119192    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb6bf60-cf44-4fff-ac7b-0995fced611e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-481cb7b8-st2cn\" (UID: \"6cb6bf60-cf44-4fff-ac7b-0995fced611e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"
Nov 02 12:12:40.121997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:40.119357    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb6bf60-cf44-4fff-ac7b-0995fced611e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-481cb7b8-st2cn\" (UID: \"6cb6bf60-cf44-4fff-ac7b-0995fced611e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"
Nov 02 12:12:40.126032 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:40.122632    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb6bf60-cf44-4fff-ac7b-0995fced611e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-481cb7b8-st2cn\" (UID: \"6cb6bf60-cf44-4fff-ac7b-0995fced611e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"
Nov 02 12:12:40.197995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:40.197047    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnbmw\" (UniqueName: \"kubernetes.io/projected/6cb6bf60-cf44-4fff-ac7b-0995fced611e-kube-api-access-qnbmw\") pod \"maas-api-revoke-sub-481cb7b8-st2cn\" (UID: \"6cb6bf60-cf44-4fff-ac7b-0995fced611e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"
Nov 02 12:12:40.200879 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:40.200840    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"
Nov 02 12:12:40.600090 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:40.600066    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"]
Nov 02 12:12:40.602213 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:12:40.602187    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb6bf60_cf44_4fff_ac7b_0995fced611e.slice/crio-b51a81af96540046d5e1b4e56714f36d43dcdd60db3f3f8a83877aecfbce6326 WatchSource:0}: Error finding container b51a81af96540046d5e1b4e56714f36d43dcdd60db3f3f8a83877aecfbce6326: Status 404 returned error can't find the container with id b51a81af96540046d5e1b4e56714f36d43dcdd60db3f3f8a83877aecfbce6326
Nov 02 12:12:40.829969 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:40.829937    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn" event={"ID":"6cb6bf60-cf44-4fff-ac7b-0995fced611e","Type":"ContainerStarted","Data":"b51a81af96540046d5e1b4e56714f36d43dcdd60db3f3f8a83877aecfbce6326"}
Nov 02 12:12:41.837450 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:41.837419    4464 generic.go:356] "Generic (PLEG): container finished" podID="6cb6bf60-cf44-4fff-ac7b-0995fced611e" containerID="41b74a0a03f7fc04b6bac1240f2bf35b998ee5845ebe3cc0d75a31b53b4fd5f2" exitCode=0
Nov 02 12:12:41.837808 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:41.837454    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn" event={"ID":"6cb6bf60-cf44-4fff-ac7b-0995fced611e","Type":"ContainerDied","Data":"41b74a0a03f7fc04b6bac1240f2bf35b998ee5845ebe3cc0d75a31b53b4fd5f2"}
Nov 02 12:12:43.008813 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:43.008789    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"
Nov 02 12:12:43.162204 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:43.162123    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb6bf60-cf44-4fff-ac7b-0995fced611e-kube-api-access-qnbmw\" (UniqueName: \"kubernetes.io/projected/6cb6bf60-cf44-4fff-ac7b-0995fced611e-kube-api-access-qnbmw\") pod \"6cb6bf60-cf44-4fff-ac7b-0995fced611e\" (UID: \"6cb6bf60-cf44-4fff-ac7b-0995fced611e\") "
Nov 02 12:12:43.162204 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:43.162166    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb6bf60-cf44-4fff-ac7b-0995fced611e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb6bf60-cf44-4fff-ac7b-0995fced611e-maas-api-openshift-ca\") pod \"6cb6bf60-cf44-4fff-ac7b-0995fced611e\" (UID: \"6cb6bf60-cf44-4fff-ac7b-0995fced611e\") "
Nov 02 12:12:43.162462 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:43.162327    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cb6bf60-cf44-4fff-ac7b-0995fced611e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb6bf60-cf44-4fff-ac7b-0995fced611e-maas-api-xks-ca\") pod \"6cb6bf60-cf44-4fff-ac7b-0995fced611e\" (UID: \"6cb6bf60-cf44-4fff-ac7b-0995fced611e\") "
Nov 02 12:12:43.162638 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:43.162603    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb6bf60-cf44-4fff-ac7b-0995fced611e-maas-api-openshift-ca" pod "6cb6bf60-cf44-4fff-ac7b-0995fced611e" (UID: "6cb6bf60-cf44-4fff-ac7b-0995fced611e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:12:43.162799 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:43.162781    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb6bf60-cf44-4fff-ac7b-0995fced611e-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:12:43.164457 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:43.164433    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb6bf60-cf44-4fff-ac7b-0995fced611e-kube-api-access-qnbmw" pod "6cb6bf60-cf44-4fff-ac7b-0995fced611e" (UID: "6cb6bf60-cf44-4fff-ac7b-0995fced611e"). InnerVolumeSpecName "kube-api-access-qnbmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:12:43.164749 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:43.164731    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb6bf60-cf44-4fff-ac7b-0995fced611e-maas-api-xks-ca" pod "6cb6bf60-cf44-4fff-ac7b-0995fced611e" (UID: "6cb6bf60-cf44-4fff-ac7b-0995fced611e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:12:43.263852 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:43.263827    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb6bf60-cf44-4fff-ac7b-0995fced611e-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:12:43.263852 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:43.263850    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnbmw\" (UniqueName: \"kubernetes.io/projected/6cb6bf60-cf44-4fff-ac7b-0995fced611e-kube-api-access-qnbmw\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:12:43.852777 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:43.852695    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"
Nov 02 12:12:43.852777 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:43.852720    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn" event={"ID":"6cb6bf60-cf44-4fff-ac7b-0995fced611e","Type":"ContainerDied","Data":"b51a81af96540046d5e1b4e56714f36d43dcdd60db3f3f8a83877aecfbce6326"}
Nov 02 12:12:43.852777 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:43.852771    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b51a81af96540046d5e1b4e56714f36d43dcdd60db3f3f8a83877aecfbce6326"
Nov 02 12:12:46.802335 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:46.802307    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"
Nov 02 12:12:46.802741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:46.802377    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"
Nov 02 12:12:48.118922 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.118886    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"]
Nov 02 12:12:48.119581 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.119561    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6cb6bf60-cf44-4fff-ac7b-0995fced611e" containerName="revoke-keys"
Nov 02 12:12:48.119629 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.119584    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb6bf60-cf44-4fff-ac7b-0995fced611e" containerName="revoke-keys"
Nov 02 12:12:48.119680 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.119671    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb6bf60-cf44-4fff-ac7b-0995fced611e" containerName="revoke-keys"
Nov 02 12:12:48.123399 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.123383    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"
Nov 02 12:12:48.180311 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.180289    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"]
Nov 02 12:12:48.223269 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.223240    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86dbb9aa-6gczb\" (UID: \"baa262bd-0aa2-4022-be8d-4c0efc3c4f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"
Nov 02 12:12:48.223393 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.223308    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2xgz\" (UniqueName: \"kubernetes.io/projected/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-kube-api-access-b2xgz\") pod \"maas-api-revoke-sub-86dbb9aa-6gczb\" (UID: \"baa262bd-0aa2-4022-be8d-4c0efc3c4f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"
Nov 02 12:12:48.223433 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.223405    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86dbb9aa-6gczb\" (UID: \"baa262bd-0aa2-4022-be8d-4c0efc3c4f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"
Nov 02 12:12:48.227244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.227229    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:12:48.227315 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.227264    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:12:48.253148 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.253128    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:12:48.253247 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.253152    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:12:48.324430 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.324400    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2xgz\" (UniqueName: \"kubernetes.io/projected/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-kube-api-access-b2xgz\") pod \"maas-api-revoke-sub-86dbb9aa-6gczb\" (UID: \"baa262bd-0aa2-4022-be8d-4c0efc3c4f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"
Nov 02 12:12:48.324546 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.324467    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86dbb9aa-6gczb\" (UID: \"baa262bd-0aa2-4022-be8d-4c0efc3c4f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"
Nov 02 12:12:48.324620 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.324605    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86dbb9aa-6gczb\" (UID: \"baa262bd-0aa2-4022-be8d-4c0efc3c4f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"
Nov 02 12:12:48.325208 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.325180    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86dbb9aa-6gczb\" (UID: \"baa262bd-0aa2-4022-be8d-4c0efc3c4f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"
Nov 02 12:12:48.326850 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.326833    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86dbb9aa-6gczb\" (UID: \"baa262bd-0aa2-4022-be8d-4c0efc3c4f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"
Nov 02 12:12:48.386772 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.386720    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2xgz\" (UniqueName: \"kubernetes.io/projected/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-kube-api-access-b2xgz\") pod \"maas-api-revoke-sub-86dbb9aa-6gczb\" (UID: \"baa262bd-0aa2-4022-be8d-4c0efc3c4f57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"
Nov 02 12:12:48.437031 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.437004    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"
Nov 02 12:12:48.814116 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.814087    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"]
Nov 02 12:12:48.814775 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:12:48.814747    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa262bd_0aa2_4022_be8d_4c0efc3c4f57.slice/crio-4836d9991cf4ad3ea416122fbb7e863f1a27d9b992489d96021e11c7cf9cf821 WatchSource:0}: Error finding container 4836d9991cf4ad3ea416122fbb7e863f1a27d9b992489d96021e11c7cf9cf821: Status 404 returned error can't find the container with id 4836d9991cf4ad3ea416122fbb7e863f1a27d9b992489d96021e11c7cf9cf821
Nov 02 12:12:48.905390 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:48.905351    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb" event={"ID":"baa262bd-0aa2-4022-be8d-4c0efc3c4f57","Type":"ContainerStarted","Data":"4836d9991cf4ad3ea416122fbb7e863f1a27d9b992489d96021e11c7cf9cf821"}
Nov 02 12:12:49.522425 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:49.522392    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"]
Nov 02 12:12:49.912571 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:49.912482    4464 generic.go:356] "Generic (PLEG): container finished" podID="baa262bd-0aa2-4022-be8d-4c0efc3c4f57" containerID="fc139f81238554f8b46dadcf71a77af423180febdf21e02bcdc493cb09b0040f" exitCode=0
Nov 02 12:12:49.912723 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:49.912572    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb" event={"ID":"baa262bd-0aa2-4022-be8d-4c0efc3c4f57","Type":"ContainerDied","Data":"fc139f81238554f8b46dadcf71a77af423180febdf21e02bcdc493cb09b0040f"}
Nov 02 12:12:51.122394 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:51.122359    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"
Nov 02 12:12:51.284244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:51.283426    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-maas-api-xks-ca\") pod \"baa262bd-0aa2-4022-be8d-4c0efc3c4f57\" (UID: \"baa262bd-0aa2-4022-be8d-4c0efc3c4f57\") "
Nov 02 12:12:51.284244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:51.283510    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-maas-api-openshift-ca\") pod \"baa262bd-0aa2-4022-be8d-4c0efc3c4f57\" (UID: \"baa262bd-0aa2-4022-be8d-4c0efc3c4f57\") "
Nov 02 12:12:51.285307 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:51.285278    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-kube-api-access-b2xgz\" (UniqueName: \"kubernetes.io/projected/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-kube-api-access-b2xgz\") pod \"baa262bd-0aa2-4022-be8d-4c0efc3c4f57\" (UID: \"baa262bd-0aa2-4022-be8d-4c0efc3c4f57\") "
Nov 02 12:12:51.285706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:51.285676    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-maas-api-openshift-ca" pod "baa262bd-0aa2-4022-be8d-4c0efc3c4f57" (UID: "baa262bd-0aa2-4022-be8d-4c0efc3c4f57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:12:51.286101 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:51.286078    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:12:51.300847 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:51.300683    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-maas-api-xks-ca" pod "baa262bd-0aa2-4022-be8d-4c0efc3c4f57" (UID: "baa262bd-0aa2-4022-be8d-4c0efc3c4f57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:12:51.300847 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:51.300813    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-kube-api-access-b2xgz" pod "baa262bd-0aa2-4022-be8d-4c0efc3c4f57" (UID: "baa262bd-0aa2-4022-be8d-4c0efc3c4f57"). InnerVolumeSpecName "kube-api-access-b2xgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:12:51.387732 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:51.387704    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2xgz\" (UniqueName: \"kubernetes.io/projected/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-kube-api-access-b2xgz\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:12:51.387732 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:51.387729    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa262bd-0aa2-4022-be8d-4c0efc3c4f57-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:12:51.929139 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:51.929063    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"
Nov 02 12:12:51.929139 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:51.929100    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb" event={"ID":"baa262bd-0aa2-4022-be8d-4c0efc3c4f57","Type":"ContainerDied","Data":"4836d9991cf4ad3ea416122fbb7e863f1a27d9b992489d96021e11c7cf9cf821"}
Nov 02 12:12:51.929327 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:51.929158    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4836d9991cf4ad3ea416122fbb7e863f1a27d9b992489d96021e11c7cf9cf821"
Nov 02 12:12:54.523456 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:54.523398    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn" podUID="18d62ecb-65b0-427f-8177-9a0fb203a358" containerName="payload-pre-processing" containerID="cri-o://62742b04853465915374c6f6272e6b3d83eda72a6a1335ea332b23bd3cd52cc0" gracePeriod=25
Nov 02 12:12:54.846219 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:54.846134    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn" podUID="18d62ecb-65b0-427f-8177-9a0fb203a358" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:12:55.083918 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.083885    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:12:55.376772 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.376741    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"]
Nov 02 12:12:55.377820 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.377796    4464 cpu_manager.go:414] "containerMap: removing container" podUID="baa262bd-0aa2-4022-be8d-4c0efc3c4f57" containerName="revoke-keys"
Nov 02 12:12:55.377965 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.377956    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baa262bd-0aa2-4022-be8d-4c0efc3c4f57" containerName="revoke-keys"
Nov 02 12:12:55.378211 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.378199    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="baa262bd-0aa2-4022-be8d-4c0efc3c4f57" containerName="revoke-keys"
Nov 02 12:12:55.382181 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.382162    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"
Nov 02 12:12:55.416825 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.416800    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"]
Nov 02 12:12:55.436963 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.436935    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:12:55.439448 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.439429    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:12:55.463686 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.463647    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c016e94-ec5b-499c-97af-e46396706a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-994ce40c-bbx6h\" (UID: \"4c016e94-ec5b-499c-97af-e46396706a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"
Nov 02 12:12:55.463944 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.463874    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c016e94-ec5b-499c-97af-e46396706a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-994ce40c-bbx6h\" (UID: \"4c016e94-ec5b-499c-97af-e46396706a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"
Nov 02 12:12:55.463944 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.463731    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:12:55.464134 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.464061    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjklr\" (UniqueName: \"kubernetes.io/projected/4c016e94-ec5b-499c-97af-e46396706a71-kube-api-access-pjklr\") pod \"maas-api-revoke-sub-994ce40c-bbx6h\" (UID: \"4c016e94-ec5b-499c-97af-e46396706a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"
Nov 02 12:12:55.487494 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.487471    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:12:55.565615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.565591    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c016e94-ec5b-499c-97af-e46396706a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-994ce40c-bbx6h\" (UID: \"4c016e94-ec5b-499c-97af-e46396706a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"
Nov 02 12:12:55.566084 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.565624    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c016e94-ec5b-499c-97af-e46396706a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-994ce40c-bbx6h\" (UID: \"4c016e94-ec5b-499c-97af-e46396706a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"
Nov 02 12:12:55.566084 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.565677    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjklr\" (UniqueName: \"kubernetes.io/projected/4c016e94-ec5b-499c-97af-e46396706a71-kube-api-access-pjklr\") pod \"maas-api-revoke-sub-994ce40c-bbx6h\" (UID: \"4c016e94-ec5b-499c-97af-e46396706a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"
Nov 02 12:12:55.566266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.566248    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c016e94-ec5b-499c-97af-e46396706a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-994ce40c-bbx6h\" (UID: \"4c016e94-ec5b-499c-97af-e46396706a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"
Nov 02 12:12:55.568045 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.568021    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c016e94-ec5b-499c-97af-e46396706a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-994ce40c-bbx6h\" (UID: \"4c016e94-ec5b-499c-97af-e46396706a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"
Nov 02 12:12:55.627381 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.627326    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjklr\" (UniqueName: \"kubernetes.io/projected/4c016e94-ec5b-499c-97af-e46396706a71-kube-api-access-pjklr\") pod \"maas-api-revoke-sub-994ce40c-bbx6h\" (UID: \"4c016e94-ec5b-499c-97af-e46396706a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"
Nov 02 12:12:55.699361 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:55.699340    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"
Nov 02 12:12:56.076072 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:56.076038    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"]
Nov 02 12:12:56.077122 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:12:56.077094    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c016e94_ec5b_499c_97af_e46396706a71.slice/crio-9991af3abc281ab57e8c32fb9038e5aeb56a2d6fcced2ef19ce5d508993f2abb WatchSource:0}: Error finding container 9991af3abc281ab57e8c32fb9038e5aeb56a2d6fcced2ef19ce5d508993f2abb: Status 404 returned error can't find the container with id 9991af3abc281ab57e8c32fb9038e5aeb56a2d6fcced2ef19ce5d508993f2abb
Nov 02 12:12:56.960910 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:56.960883    4464 generic.go:356] "Generic (PLEG): container finished" podID="4c016e94-ec5b-499c-97af-e46396706a71" containerID="38d832646261daa3aa56688202e407311bf9e8ffec9ee7d2ab435acbd659795e" exitCode=0
Nov 02 12:12:56.961264 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:56.960936    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h" event={"ID":"4c016e94-ec5b-499c-97af-e46396706a71","Type":"ContainerDied","Data":"38d832646261daa3aa56688202e407311bf9e8ffec9ee7d2ab435acbd659795e"}
Nov 02 12:12:56.961264 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:56.961003    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h" event={"ID":"4c016e94-ec5b-499c-97af-e46396706a71","Type":"ContainerStarted","Data":"9991af3abc281ab57e8c32fb9038e5aeb56a2d6fcced2ef19ce5d508993f2abb"}
Nov 02 12:12:58.107026 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:58.106970    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"
Nov 02 12:12:58.194718 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:58.194695    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c016e94-ec5b-499c-97af-e46396706a71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c016e94-ec5b-499c-97af-e46396706a71-maas-api-xks-ca\") pod \"4c016e94-ec5b-499c-97af-e46396706a71\" (UID: \"4c016e94-ec5b-499c-97af-e46396706a71\") "
Nov 02 12:12:58.194848 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:58.194807    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c016e94-ec5b-499c-97af-e46396706a71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c016e94-ec5b-499c-97af-e46396706a71-maas-api-openshift-ca\") pod \"4c016e94-ec5b-499c-97af-e46396706a71\" (UID: \"4c016e94-ec5b-499c-97af-e46396706a71\") "
Nov 02 12:12:58.194848 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:58.194830    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c016e94-ec5b-499c-97af-e46396706a71-kube-api-access-pjklr\" (UniqueName: \"kubernetes.io/projected/4c016e94-ec5b-499c-97af-e46396706a71-kube-api-access-pjklr\") pod \"4c016e94-ec5b-499c-97af-e46396706a71\" (UID: \"4c016e94-ec5b-499c-97af-e46396706a71\") "
Nov 02 12:12:58.195174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:58.195152    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c016e94-ec5b-499c-97af-e46396706a71-maas-api-openshift-ca" pod "4c016e94-ec5b-499c-97af-e46396706a71" (UID: "4c016e94-ec5b-499c-97af-e46396706a71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:12:58.196710 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:58.196680    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c016e94-ec5b-499c-97af-e46396706a71-maas-api-xks-ca" pod "4c016e94-ec5b-499c-97af-e46396706a71" (UID: "4c016e94-ec5b-499c-97af-e46396706a71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:12:58.196793 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:58.196751    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c016e94-ec5b-499c-97af-e46396706a71-kube-api-access-pjklr" pod "4c016e94-ec5b-499c-97af-e46396706a71" (UID: "4c016e94-ec5b-499c-97af-e46396706a71"). InnerVolumeSpecName "kube-api-access-pjklr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:12:58.291849 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:12:58.291814    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:12:58.296606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:58.296577    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c016e94-ec5b-499c-97af-e46396706a71-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:12:58.296606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:58.296604    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjklr\" (UniqueName: \"kubernetes.io/projected/4c016e94-ec5b-499c-97af-e46396706a71-kube-api-access-pjklr\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:12:58.296818 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:58.296615    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c016e94-ec5b-499c-97af-e46396706a71-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:12:58.974736 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:58.974707    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"
Nov 02 12:12:58.974905 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:58.974740    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h" event={"ID":"4c016e94-ec5b-499c-97af-e46396706a71","Type":"ContainerDied","Data":"9991af3abc281ab57e8c32fb9038e5aeb56a2d6fcced2ef19ce5d508993f2abb"}
Nov 02 12:12:58.974905 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:58.974791    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9991af3abc281ab57e8c32fb9038e5aeb56a2d6fcced2ef19ce5d508993f2abb"
Nov 02 12:12:59.094046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.094013    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-vbstg-xx28x"]
Nov 02 12:12:59.094663 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.094644    4464 cpu_manager.go:414] "containerMap: removing container" podUID="4c016e94-ec5b-499c-97af-e46396706a71" containerName="revoke-keys"
Nov 02 12:12:59.094663 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.094660    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c016e94-ec5b-499c-97af-e46396706a71" containerName="revoke-keys"
Nov 02 12:12:59.094841 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.094763    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c016e94-ec5b-499c-97af-e46396706a71" containerName="revoke-keys"
Nov 02 12:12:59.100052 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.100034    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vbstg-xx28x"
Nov 02 12:12:59.153502 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.153476    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 12:12:59.154939 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.154921    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vbstg-xx28x"]
Nov 02 12:12:59.202049 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.202023    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vv2bx\""
Nov 02 12:12:59.205683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.205666    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52936383-9095-430a-9f24-6b8a649ee7c8-trusted-ca-bundle\") pod \"periodic-gathering-vbstg-xx28x\" (UID: \"52936383-9095-430a-9f24-6b8a649ee7c8\") " pod="openshift-insights/periodic-gathering-vbstg-xx28x"
Nov 02 12:12:59.205803 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.205787    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52936383-9095-430a-9f24-6b8a649ee7c8-service-ca-bundle\") pod \"periodic-gathering-vbstg-xx28x\" (UID: \"52936383-9095-430a-9f24-6b8a649ee7c8\") " pod="openshift-insights/periodic-gathering-vbstg-xx28x"
Nov 02 12:12:59.205930 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.205914    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6spj4\" (UniqueName: \"kubernetes.io/projected/52936383-9095-430a-9f24-6b8a649ee7c8-kube-api-access-6spj4\") pod \"periodic-gathering-vbstg-xx28x\" (UID: \"52936383-9095-430a-9f24-6b8a649ee7c8\") " pod="openshift-insights/periodic-gathering-vbstg-xx28x"
Nov 02 12:12:59.205994 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.205970    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/52936383-9095-430a-9f24-6b8a649ee7c8-archives-path\") pod \"periodic-gathering-vbstg-xx28x\" (UID: \"52936383-9095-430a-9f24-6b8a649ee7c8\") " pod="openshift-insights/periodic-gathering-vbstg-xx28x"
Nov 02 12:12:59.304328 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.304261    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 12:12:59.306812 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.306783    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6spj4\" (UniqueName: \"kubernetes.io/projected/52936383-9095-430a-9f24-6b8a649ee7c8-kube-api-access-6spj4\") pod \"periodic-gathering-vbstg-xx28x\" (UID: \"52936383-9095-430a-9f24-6b8a649ee7c8\") " pod="openshift-insights/periodic-gathering-vbstg-xx28x"
Nov 02 12:12:59.306936 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.306856    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/52936383-9095-430a-9f24-6b8a649ee7c8-archives-path\") pod \"periodic-gathering-vbstg-xx28x\" (UID: \"52936383-9095-430a-9f24-6b8a649ee7c8\") " pod="openshift-insights/periodic-gathering-vbstg-xx28x"
Nov 02 12:12:59.306936 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.306887    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52936383-9095-430a-9f24-6b8a649ee7c8-trusted-ca-bundle\") pod \"periodic-gathering-vbstg-xx28x\" (UID: \"52936383-9095-430a-9f24-6b8a649ee7c8\") " pod="openshift-insights/periodic-gathering-vbstg-xx28x"
Nov 02 12:12:59.307078 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.307016    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52936383-9095-430a-9f24-6b8a649ee7c8-service-ca-bundle\") pod \"periodic-gathering-vbstg-xx28x\" (UID: \"52936383-9095-430a-9f24-6b8a649ee7c8\") " pod="openshift-insights/periodic-gathering-vbstg-xx28x"
Nov 02 12:12:59.307285 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.307265    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/52936383-9095-430a-9f24-6b8a649ee7c8-archives-path\") pod \"periodic-gathering-vbstg-xx28x\" (UID: \"52936383-9095-430a-9f24-6b8a649ee7c8\") " pod="openshift-insights/periodic-gathering-vbstg-xx28x"
Nov 02 12:12:59.307723 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.307698    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52936383-9095-430a-9f24-6b8a649ee7c8-service-ca-bundle\") pod \"periodic-gathering-vbstg-xx28x\" (UID: \"52936383-9095-430a-9f24-6b8a649ee7c8\") " pod="openshift-insights/periodic-gathering-vbstg-xx28x"
Nov 02 12:12:59.308442 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.308420    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52936383-9095-430a-9f24-6b8a649ee7c8-trusted-ca-bundle\") pod \"periodic-gathering-vbstg-xx28x\" (UID: \"52936383-9095-430a-9f24-6b8a649ee7c8\") " pod="openshift-insights/periodic-gathering-vbstg-xx28x"
Nov 02 12:12:59.369698 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.369650    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6spj4\" (UniqueName: \"kubernetes.io/projected/52936383-9095-430a-9f24-6b8a649ee7c8-kube-api-access-6spj4\") pod \"periodic-gathering-vbstg-xx28x\" (UID: \"52936383-9095-430a-9f24-6b8a649ee7c8\") " pod="openshift-insights/periodic-gathering-vbstg-xx28x"
Nov 02 12:12:59.412717 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.412694    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vbstg-xx28x"
Nov 02 12:12:59.790195 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.790168    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vbstg-xx28x"]
Nov 02 12:12:59.791373 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:12:59.791343    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52936383_9095_430a_9f24_6b8a649ee7c8.slice/crio-9207de42c8392fbaac63daf5c1e194ab1f73e526045bf1ac7fe1b55f45d12095 WatchSource:0}: Error finding container 9207de42c8392fbaac63daf5c1e194ab1f73e526045bf1ac7fe1b55f45d12095: Status 404 returned error can't find the container with id 9207de42c8392fbaac63daf5c1e194ab1f73e526045bf1ac7fe1b55f45d12095
Nov 02 12:12:59.983186 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.983149    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vbstg-xx28x" event={"ID":"52936383-9095-430a-9f24-6b8a649ee7c8","Type":"ContainerStarted","Data":"d8803c291f106aa35b77c6d6fb85fe6179ece3839b2aa291890e362ae15b3f58"}
Nov 02 12:12:59.983332 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:12:59.983194    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vbstg-xx28x" event={"ID":"52936383-9095-430a-9f24-6b8a649ee7c8","Type":"ContainerStarted","Data":"9207de42c8392fbaac63daf5c1e194ab1f73e526045bf1ac7fe1b55f45d12095"}
Nov 02 12:13:00.104014 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:00.103904    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-vbstg-xx28x" podStartSLOduration=1.103892032 podStartE2EDuration="1.103892032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:13:00.102297805 +0000 UTC m=+1008.666006216" watchObservedRunningTime="2026-11-02 12:13:00.103892032 +0000 UTC m=+1008.667600440"
Nov 02 12:13:04.846211 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:04.846171    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn" podUID="18d62ecb-65b0-427f-8177-9a0fb203a358" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:13:11.264180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.264139    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"]
Nov 02 12:13:11.269679 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.269656    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"
Nov 02 12:13:11.323495 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.323261    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:13:11.326426 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.326393    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"]
Nov 02 12:13:11.371559 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.371525    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:13:11.371766 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.371624    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:13:11.372813 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.372786    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqzfx\" (UniqueName: \"kubernetes.io/projected/287bc5af-fcfd-4d80-ae6a-2325c63c0596-kube-api-access-jqzfx\") pod \"maas-api-revoke-sub-3d8cc74d-4fxvj\" (UID: \"287bc5af-fcfd-4d80-ae6a-2325c63c0596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"
Nov 02 12:13:11.373117 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.373094    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/287bc5af-fcfd-4d80-ae6a-2325c63c0596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d8cc74d-4fxvj\" (UID: \"287bc5af-fcfd-4d80-ae6a-2325c63c0596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"
Nov 02 12:13:11.373226 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.373166    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/287bc5af-fcfd-4d80-ae6a-2325c63c0596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d8cc74d-4fxvj\" (UID: \"287bc5af-fcfd-4d80-ae6a-2325c63c0596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"
Nov 02 12:13:11.373816 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.373793    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:13:11.474031 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.473968    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/287bc5af-fcfd-4d80-ae6a-2325c63c0596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d8cc74d-4fxvj\" (UID: \"287bc5af-fcfd-4d80-ae6a-2325c63c0596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"
Nov 02 12:13:11.474226 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.474043    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/287bc5af-fcfd-4d80-ae6a-2325c63c0596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d8cc74d-4fxvj\" (UID: \"287bc5af-fcfd-4d80-ae6a-2325c63c0596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"
Nov 02 12:13:11.474226 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.474132    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqzfx\" (UniqueName: \"kubernetes.io/projected/287bc5af-fcfd-4d80-ae6a-2325c63c0596-kube-api-access-jqzfx\") pod \"maas-api-revoke-sub-3d8cc74d-4fxvj\" (UID: \"287bc5af-fcfd-4d80-ae6a-2325c63c0596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"
Nov 02 12:13:11.475196 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.475164    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/287bc5af-fcfd-4d80-ae6a-2325c63c0596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d8cc74d-4fxvj\" (UID: \"287bc5af-fcfd-4d80-ae6a-2325c63c0596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"
Nov 02 12:13:11.476570 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.476543    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/287bc5af-fcfd-4d80-ae6a-2325c63c0596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d8cc74d-4fxvj\" (UID: \"287bc5af-fcfd-4d80-ae6a-2325c63c0596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"
Nov 02 12:13:11.536024 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.535916    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqzfx\" (UniqueName: \"kubernetes.io/projected/287bc5af-fcfd-4d80-ae6a-2325c63c0596-kube-api-access-jqzfx\") pod \"maas-api-revoke-sub-3d8cc74d-4fxvj\" (UID: \"287bc5af-fcfd-4d80-ae6a-2325c63c0596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"
Nov 02 12:13:11.640302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.640267    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:13:11.649635 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:11.649594    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"
Nov 02 12:13:12.262742 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:12.262686    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"]
Nov 02 12:13:13.143912 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:13.143818    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj" event={"ID":"287bc5af-fcfd-4d80-ae6a-2325c63c0596","Type":"ContainerStarted","Data":"ab7cbdac041ad1d49affae1d4c1475e6660e41799930d2875299123c1ba41404"}
Nov 02 12:13:13.143912 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:13.143871    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj" event={"ID":"287bc5af-fcfd-4d80-ae6a-2325c63c0596","Type":"ContainerStarted","Data":"d0fbd06cead385ee4d645c3cadd2630a943217c338d19283c92571c8b6cb7377"}
Nov 02 12:13:13.263162 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:13.263117    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj" podStartSLOduration=2.2631032380000002 podStartE2EDuration="2.263103238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:13:13.262241769 +0000 UTC m=+1021.825950211" watchObservedRunningTime="2026-11-02 12:13:13.263103238 +0000 UTC m=+1021.826811647"
Nov 02 12:13:14.150997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:14.150941    4464 generic.go:356] "Generic (PLEG): container finished" podID="287bc5af-fcfd-4d80-ae6a-2325c63c0596" containerID="ab7cbdac041ad1d49affae1d4c1475e6660e41799930d2875299123c1ba41404" exitCode=0
Nov 02 12:13:14.151354 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:14.151022    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj" event={"ID":"287bc5af-fcfd-4d80-ae6a-2325c63c0596","Type":"ContainerDied","Data":"ab7cbdac041ad1d49affae1d4c1475e6660e41799930d2875299123c1ba41404"}
Nov 02 12:13:14.859159 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:14.859133    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"
Nov 02 12:13:14.959897 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:14.959854    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d62ecb-65b0-427f-8177-9a0fb203a358-kube-api-access-p8vjn\" (UniqueName: \"kubernetes.io/projected/18d62ecb-65b0-427f-8177-9a0fb203a358-kube-api-access-p8vjn\") pod \"18d62ecb-65b0-427f-8177-9a0fb203a358\" (UID: \"18d62ecb-65b0-427f-8177-9a0fb203a358\") "
Nov 02 12:13:14.960125 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:14.959919    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18d62ecb-65b0-427f-8177-9a0fb203a358-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d62ecb-65b0-427f-8177-9a0fb203a358-plugins-config-volume\") pod \"18d62ecb-65b0-427f-8177-9a0fb203a358\" (UID: \"18d62ecb-65b0-427f-8177-9a0fb203a358\") "
Nov 02 12:13:14.960313 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:14.960287    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d62ecb-65b0-427f-8177-9a0fb203a358-plugins-config-volume" pod "18d62ecb-65b0-427f-8177-9a0fb203a358" (UID: "18d62ecb-65b0-427f-8177-9a0fb203a358"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:13:14.960540 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:14.960517    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18d62ecb-65b0-427f-8177-9a0fb203a358-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:13:14.961952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:14.961929    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d62ecb-65b0-427f-8177-9a0fb203a358-kube-api-access-p8vjn" pod "18d62ecb-65b0-427f-8177-9a0fb203a358" (UID: "18d62ecb-65b0-427f-8177-9a0fb203a358"). InnerVolumeSpecName "kube-api-access-p8vjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:13:15.070048 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.065998    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8vjn\" (UniqueName: \"kubernetes.io/projected/18d62ecb-65b0-427f-8177-9a0fb203a358-kube-api-access-p8vjn\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:13:15.158807 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.158711    4464 generic.go:356] "Generic (PLEG): container finished" podID="18d62ecb-65b0-427f-8177-9a0fb203a358" containerID="62742b04853465915374c6f6272e6b3d83eda72a6a1335ea332b23bd3cd52cc0" exitCode=0
Nov 02 12:13:15.158807 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.158761    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"
Nov 02 12:13:15.158807 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.158792    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn" event={"ID":"18d62ecb-65b0-427f-8177-9a0fb203a358","Type":"ContainerDied","Data":"62742b04853465915374c6f6272e6b3d83eda72a6a1335ea332b23bd3cd52cc0"}
Nov 02 12:13:15.159388 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.158851    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn" event={"ID":"18d62ecb-65b0-427f-8177-9a0fb203a358","Type":"ContainerDied","Data":"787517ddc0fe03541538c07f88e689d843ae416d3df1322adffba059acae548e"}
Nov 02 12:13:15.159388 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.158884    4464 scope.go:122] "RemoveContainer" containerID="62742b04853465915374c6f6272e6b3d83eda72a6a1335ea332b23bd3cd52cc0"
Nov 02 12:13:15.174293 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.174275    4464 scope.go:122] "RemoveContainer" containerID="62742b04853465915374c6f6272e6b3d83eda72a6a1335ea332b23bd3cd52cc0"
Nov 02 12:13:15.174645 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:13:15.174608    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62742b04853465915374c6f6272e6b3d83eda72a6a1335ea332b23bd3cd52cc0\": container with ID starting with 62742b04853465915374c6f6272e6b3d83eda72a6a1335ea332b23bd3cd52cc0 not found: ID does not exist" containerID="62742b04853465915374c6f6272e6b3d83eda72a6a1335ea332b23bd3cd52cc0"
Nov 02 12:13:15.174727 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.174650    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62742b04853465915374c6f6272e6b3d83eda72a6a1335ea332b23bd3cd52cc0"} err="failed to get container status \"62742b04853465915374c6f6272e6b3d83eda72a6a1335ea332b23bd3cd52cc0\": rpc error: code = NotFound desc = could not find container \"62742b04853465915374c6f6272e6b3d83eda72a6a1335ea332b23bd3cd52cc0\": container with ID starting with 62742b04853465915374c6f6272e6b3d83eda72a6a1335ea332b23bd3cd52cc0 not found: ID does not exist"
Nov 02 12:13:15.296759 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.296730    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"
Nov 02 12:13:15.333201 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.333173    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"]
Nov 02 12:13:15.337132 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.337109    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn"]
Nov 02 12:13:15.384184 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.384153    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/287bc5af-fcfd-4d80-ae6a-2325c63c0596-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/287bc5af-fcfd-4d80-ae6a-2325c63c0596-maas-api-xks-ca\") pod \"287bc5af-fcfd-4d80-ae6a-2325c63c0596\" (UID: \"287bc5af-fcfd-4d80-ae6a-2325c63c0596\") "
Nov 02 12:13:15.384349 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.384261    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/287bc5af-fcfd-4d80-ae6a-2325c63c0596-kube-api-access-jqzfx\" (UniqueName: \"kubernetes.io/projected/287bc5af-fcfd-4d80-ae6a-2325c63c0596-kube-api-access-jqzfx\") pod \"287bc5af-fcfd-4d80-ae6a-2325c63c0596\" (UID: \"287bc5af-fcfd-4d80-ae6a-2325c63c0596\") "
Nov 02 12:13:15.384349 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.384329    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/287bc5af-fcfd-4d80-ae6a-2325c63c0596-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/287bc5af-fcfd-4d80-ae6a-2325c63c0596-maas-api-openshift-ca\") pod \"287bc5af-fcfd-4d80-ae6a-2325c63c0596\" (UID: \"287bc5af-fcfd-4d80-ae6a-2325c63c0596\") "
Nov 02 12:13:15.384707 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.384684    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/287bc5af-fcfd-4d80-ae6a-2325c63c0596-maas-api-openshift-ca" pod "287bc5af-fcfd-4d80-ae6a-2325c63c0596" (UID: "287bc5af-fcfd-4d80-ae6a-2325c63c0596"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:13:15.386266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.386246    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/287bc5af-fcfd-4d80-ae6a-2325c63c0596-maas-api-xks-ca" pod "287bc5af-fcfd-4d80-ae6a-2325c63c0596" (UID: "287bc5af-fcfd-4d80-ae6a-2325c63c0596"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:13:15.386332 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.386248    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287bc5af-fcfd-4d80-ae6a-2325c63c0596-kube-api-access-jqzfx" pod "287bc5af-fcfd-4d80-ae6a-2325c63c0596" (UID: "287bc5af-fcfd-4d80-ae6a-2325c63c0596"). InnerVolumeSpecName "kube-api-access-jqzfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:13:15.485434 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.485391    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/287bc5af-fcfd-4d80-ae6a-2325c63c0596-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:13:15.485434 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.485435    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/287bc5af-fcfd-4d80-ae6a-2325c63c0596-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:13:15.485684 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.485446    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqzfx\" (UniqueName: \"kubernetes.io/projected/287bc5af-fcfd-4d80-ae6a-2325c63c0596-kube-api-access-jqzfx\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:13:15.589160 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.589110    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d62ecb-65b0-427f-8177-9a0fb203a358" path="/var/lib/kubelet/pods/18d62ecb-65b0-427f-8177-9a0fb203a358/volumes"
Nov 02 12:13:15.859052 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:15.850277    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b6f3d8-578458c5f5-wrqgn" podUID="18d62ecb-65b0-427f-8177-9a0fb203a358" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.47:50052\" within 1s: context deadline exceeded"
Nov 02 12:13:15.859052 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:13:15.850506    4464 logging.go:55] [core] [Channel #935 SubChannel #936]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.47:50052", ServerName: "10.133.0.47:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.47:50052: operation was canceled"
Nov 02 12:13:16.166952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:16.166873    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"
Nov 02 12:13:16.167354 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:16.166878    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj" event={"ID":"287bc5af-fcfd-4d80-ae6a-2325c63c0596","Type":"ContainerDied","Data":"d0fbd06cead385ee4d645c3cadd2630a943217c338d19283c92571c8b6cb7377"}
Nov 02 12:13:16.167354 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:16.166999    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0fbd06cead385ee4d645c3cadd2630a943217c338d19283c92571c8b6cb7377"
Nov 02 12:13:23.626317 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:23.626283    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:13:28.322817 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:13:28.322728    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:13:31.465227 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.465193    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"]
Nov 02 12:13:31.465823 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.465808    4464 cpu_manager.go:414] "containerMap: removing container" podUID="18d62ecb-65b0-427f-8177-9a0fb203a358" containerName="payload-pre-processing"
Nov 02 12:13:31.465866 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.465824    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d62ecb-65b0-427f-8177-9a0fb203a358" containerName="payload-pre-processing"
Nov 02 12:13:31.465866 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.465842    4464 cpu_manager.go:414] "containerMap: removing container" podUID="287bc5af-fcfd-4d80-ae6a-2325c63c0596" containerName="revoke-keys"
Nov 02 12:13:31.465866 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.465848    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287bc5af-fcfd-4d80-ae6a-2325c63c0596" containerName="revoke-keys"
Nov 02 12:13:31.465959 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.465946    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d62ecb-65b0-427f-8177-9a0fb203a358" containerName="payload-pre-processing"
Nov 02 12:13:31.465959 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.465958    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="287bc5af-fcfd-4d80-ae6a-2325c63c0596" containerName="revoke-keys"
Nov 02 12:13:31.470920 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.470903    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"
Nov 02 12:13:31.527302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.527247    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"]
Nov 02 12:13:31.555294 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.555264    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-922v2\" (UniqueName: \"kubernetes.io/projected/b49c47f0-83dc-4fff-b9ed-01136280bbfe-kube-api-access-922v2\") pod \"maas-api-revoke-sub-95a16573-mvj9z\" (UID: \"b49c47f0-83dc-4fff-b9ed-01136280bbfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"
Nov 02 12:13:31.555453 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.555317    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c47f0-83dc-4fff-b9ed-01136280bbfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95a16573-mvj9z\" (UID: \"b49c47f0-83dc-4fff-b9ed-01136280bbfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"
Nov 02 12:13:31.555453 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.555336    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c47f0-83dc-4fff-b9ed-01136280bbfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95a16573-mvj9z\" (UID: \"b49c47f0-83dc-4fff-b9ed-01136280bbfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"
Nov 02 12:13:31.573760 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.573734    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:13:31.574020 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.574001    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:13:31.574093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.574072    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:13:31.574132 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.574117    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:13:31.657123 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.657088    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-922v2\" (UniqueName: \"kubernetes.io/projected/b49c47f0-83dc-4fff-b9ed-01136280bbfe-kube-api-access-922v2\") pod \"maas-api-revoke-sub-95a16573-mvj9z\" (UID: \"b49c47f0-83dc-4fff-b9ed-01136280bbfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"
Nov 02 12:13:31.657323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.657154    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c47f0-83dc-4fff-b9ed-01136280bbfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95a16573-mvj9z\" (UID: \"b49c47f0-83dc-4fff-b9ed-01136280bbfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"
Nov 02 12:13:31.657323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.657177    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c47f0-83dc-4fff-b9ed-01136280bbfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95a16573-mvj9z\" (UID: \"b49c47f0-83dc-4fff-b9ed-01136280bbfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"
Nov 02 12:13:31.657826 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.657804    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c47f0-83dc-4fff-b9ed-01136280bbfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95a16573-mvj9z\" (UID: \"b49c47f0-83dc-4fff-b9ed-01136280bbfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"
Nov 02 12:13:31.659707 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.659677    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c47f0-83dc-4fff-b9ed-01136280bbfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95a16573-mvj9z\" (UID: \"b49c47f0-83dc-4fff-b9ed-01136280bbfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"
Nov 02 12:13:31.718646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.718624    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-922v2\" (UniqueName: \"kubernetes.io/projected/b49c47f0-83dc-4fff-b9ed-01136280bbfe-kube-api-access-922v2\") pod \"maas-api-revoke-sub-95a16573-mvj9z\" (UID: \"b49c47f0-83dc-4fff-b9ed-01136280bbfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"
Nov 02 12:13:31.785287 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:31.785261    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"
Nov 02 12:13:32.192878 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:32.190040    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"]
Nov 02 12:13:32.322102 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:32.322071    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z" event={"ID":"b49c47f0-83dc-4fff-b9ed-01136280bbfe","Type":"ContainerStarted","Data":"15664ed32286b01f0d8c962ed7b0e79ea840c75716b8d5804c7664eb77fb264f"}
Nov 02 12:13:33.330095 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:33.330010    4464 generic.go:356] "Generic (PLEG): container finished" podID="b49c47f0-83dc-4fff-b9ed-01136280bbfe" containerID="46062b67fd056d75a348df435d04d7e396b5ab12f07b038afd59198d2625eba6" exitCode=0
Nov 02 12:13:33.330455 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:33.330093    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z" event={"ID":"b49c47f0-83dc-4fff-b9ed-01136280bbfe","Type":"ContainerDied","Data":"46062b67fd056d75a348df435d04d7e396b5ab12f07b038afd59198d2625eba6"}
Nov 02 12:13:34.528780 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:34.528750    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"
Nov 02 12:13:34.701443 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:34.701346    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b49c47f0-83dc-4fff-b9ed-01136280bbfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c47f0-83dc-4fff-b9ed-01136280bbfe-maas-api-openshift-ca\") pod \"b49c47f0-83dc-4fff-b9ed-01136280bbfe\" (UID: \"b49c47f0-83dc-4fff-b9ed-01136280bbfe\") "
Nov 02 12:13:34.701601 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:34.701475    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b49c47f0-83dc-4fff-b9ed-01136280bbfe-kube-api-access-922v2\" (UniqueName: \"kubernetes.io/projected/b49c47f0-83dc-4fff-b9ed-01136280bbfe-kube-api-access-922v2\") pod \"b49c47f0-83dc-4fff-b9ed-01136280bbfe\" (UID: \"b49c47f0-83dc-4fff-b9ed-01136280bbfe\") "
Nov 02 12:13:34.701601 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:34.701552    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b49c47f0-83dc-4fff-b9ed-01136280bbfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c47f0-83dc-4fff-b9ed-01136280bbfe-maas-api-xks-ca\") pod \"b49c47f0-83dc-4fff-b9ed-01136280bbfe\" (UID: \"b49c47f0-83dc-4fff-b9ed-01136280bbfe\") "
Nov 02 12:13:34.703115 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:34.703087    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b49c47f0-83dc-4fff-b9ed-01136280bbfe-maas-api-openshift-ca" pod "b49c47f0-83dc-4fff-b9ed-01136280bbfe" (UID: "b49c47f0-83dc-4fff-b9ed-01136280bbfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:13:34.726795 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:34.726476    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b49c47f0-83dc-4fff-b9ed-01136280bbfe-kube-api-access-922v2" pod "b49c47f0-83dc-4fff-b9ed-01136280bbfe" (UID: "b49c47f0-83dc-4fff-b9ed-01136280bbfe"). InnerVolumeSpecName "kube-api-access-922v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:13:34.726795 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:34.726715    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b49c47f0-83dc-4fff-b9ed-01136280bbfe-maas-api-xks-ca" pod "b49c47f0-83dc-4fff-b9ed-01136280bbfe" (UID: "b49c47f0-83dc-4fff-b9ed-01136280bbfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:13:34.808959 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:34.808879    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b49c47f0-83dc-4fff-b9ed-01136280bbfe-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:13:34.808959 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:34.808920    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-922v2\" (UniqueName: \"kubernetes.io/projected/b49c47f0-83dc-4fff-b9ed-01136280bbfe-kube-api-access-922v2\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:13:34.808959 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:34.808936    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b49c47f0-83dc-4fff-b9ed-01136280bbfe-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:13:35.346493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:35.346465    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"
Nov 02 12:13:35.346689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:35.346468    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z" event={"ID":"b49c47f0-83dc-4fff-b9ed-01136280bbfe","Type":"ContainerDied","Data":"15664ed32286b01f0d8c962ed7b0e79ea840c75716b8d5804c7664eb77fb264f"}
Nov 02 12:13:35.346689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:35.346590    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15664ed32286b01f0d8c962ed7b0e79ea840c75716b8d5804c7664eb77fb264f"
Nov 02 12:13:36.477069 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.477032    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"]
Nov 02 12:13:36.478012 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.477989    4464 cpu_manager.go:414] "containerMap: removing container" podUID="b49c47f0-83dc-4fff-b9ed-01136280bbfe" containerName="revoke-keys"
Nov 02 12:13:36.478012 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.478012    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b49c47f0-83dc-4fff-b9ed-01136280bbfe" containerName="revoke-keys"
Nov 02 12:13:36.478217 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.478201    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="b49c47f0-83dc-4fff-b9ed-01136280bbfe" containerName="revoke-keys"
Nov 02 12:13:36.482125 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.482106    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"
Nov 02 12:13:36.527590 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.527565    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr\" (UID: \"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"
Nov 02 12:13:36.527753 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.527623    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr\" (UID: \"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"
Nov 02 12:13:36.527753 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.527725    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wglz\" (UniqueName: \"kubernetes.io/projected/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-kube-api-access-2wglz\") pod \"maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr\" (UID: \"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"
Nov 02 12:13:36.535497 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.535465    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:13:36.586031 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.586006    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:13:36.586162 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.586061    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:13:36.586162 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.586061    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:13:36.589446 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.589423    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"]
Nov 02 12:13:36.629060 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.629027    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr\" (UID: \"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"
Nov 02 12:13:36.629196 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.629181    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr\" (UID: \"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"
Nov 02 12:13:36.629242 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.629211    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wglz\" (UniqueName: \"kubernetes.io/projected/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-kube-api-access-2wglz\") pod \"maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr\" (UID: \"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"
Nov 02 12:13:36.629872 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.629841    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr\" (UID: \"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"
Nov 02 12:13:36.631516 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.631494    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr\" (UID: \"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"
Nov 02 12:13:36.690790 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.690763    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wglz\" (UniqueName: \"kubernetes.io/projected/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-kube-api-access-2wglz\") pod \"maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr\" (UID: \"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"
Nov 02 12:13:36.796073 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:36.796012    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"
Nov 02 12:13:37.173970 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:37.173882    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"]
Nov 02 12:13:37.182369 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:13:37.182333    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1fc565_c18d_47bb_a6f2_e3fe68afd8a9.slice/crio-dcdaaf7f94bdd124bbf7a6abe5e8f9781c37fcabda2d636c211c09c34ec23dcc WatchSource:0}: Error finding container dcdaaf7f94bdd124bbf7a6abe5e8f9781c37fcabda2d636c211c09c34ec23dcc: Status 404 returned error can't find the container with id dcdaaf7f94bdd124bbf7a6abe5e8f9781c37fcabda2d636c211c09c34ec23dcc
Nov 02 12:13:37.362164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:37.362129    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr" event={"ID":"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9","Type":"ContainerStarted","Data":"dcdaaf7f94bdd124bbf7a6abe5e8f9781c37fcabda2d636c211c09c34ec23dcc"}
Nov 02 12:13:38.370531 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:38.370453    4464 generic.go:356] "Generic (PLEG): container finished" podID="ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9" containerID="cd0f78235ba82900bfbd187072de3fb67d93bf61c9ab232dc6a78b48cc6748d3" exitCode=0
Nov 02 12:13:38.370895 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:38.370527    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr" event={"ID":"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9","Type":"ContainerDied","Data":"cd0f78235ba82900bfbd187072de3fb67d93bf61c9ab232dc6a78b48cc6748d3"}
Nov 02 12:13:39.518832 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:39.518807    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"
Nov 02 12:13:39.560195 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:39.560170    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-maas-api-xks-ca\") pod \"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9\" (UID: \"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9\") "
Nov 02 12:13:39.560308 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:39.560272    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-maas-api-openshift-ca\") pod \"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9\" (UID: \"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9\") "
Nov 02 12:13:39.560360 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:39.560347    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-kube-api-access-2wglz\" (UniqueName: \"kubernetes.io/projected/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-kube-api-access-2wglz\") pod \"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9\" (UID: \"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9\") "
Nov 02 12:13:39.560638 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:39.560607    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-maas-api-openshift-ca" pod "ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9" (UID: "ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:13:39.562285 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:39.562257    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-kube-api-access-2wglz" pod "ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9" (UID: "ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9"). InnerVolumeSpecName "kube-api-access-2wglz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:13:39.562383 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:39.562289    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-maas-api-xks-ca" pod "ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9" (UID: "ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:13:39.662206 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:39.662076    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:13:39.662206 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:39.662114    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wglz\" (UniqueName: \"kubernetes.io/projected/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-kube-api-access-2wglz\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:13:39.662206 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:39.662131    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:13:40.384923 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:40.384895    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"
Nov 02 12:13:40.384923 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:40.384913    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr" event={"ID":"ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9","Type":"ContainerDied","Data":"dcdaaf7f94bdd124bbf7a6abe5e8f9781c37fcabda2d636c211c09c34ec23dcc"}
Nov 02 12:13:40.385163 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:40.384999    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcdaaf7f94bdd124bbf7a6abe5e8f9781c37fcabda2d636c211c09c34ec23dcc"
Nov 02 12:13:42.185895 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.185865    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"]
Nov 02 12:13:42.186485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.186470    4464 cpu_manager.go:414] "containerMap: removing container" podUID="ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9" containerName="revoke-keys"
Nov 02 12:13:42.186529 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.186486    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9" containerName="revoke-keys"
Nov 02 12:13:42.186633 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.186623    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9" containerName="revoke-keys"
Nov 02 12:13:42.190463 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.190442    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"
Nov 02 12:13:42.246905 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.246878    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"]
Nov 02 12:13:42.293565 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.293536    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:13:42.293740 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.293648    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:13:42.293740 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.293709    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:13:42.293830 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.293786    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a738826-ed52-4e69-8623-04409ac092df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-047c00af-z7nq8\" (UID: \"4a738826-ed52-4e69-8623-04409ac092df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"
Nov 02 12:13:42.294018 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.293998    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a738826-ed52-4e69-8623-04409ac092df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-047c00af-z7nq8\" (UID: \"4a738826-ed52-4e69-8623-04409ac092df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"
Nov 02 12:13:42.294081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.294062    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75wxb\" (UniqueName: \"kubernetes.io/projected/4a738826-ed52-4e69-8623-04409ac092df-kube-api-access-75wxb\") pod \"maas-api-revoke-sub-047c00af-z7nq8\" (UID: \"4a738826-ed52-4e69-8623-04409ac092df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"
Nov 02 12:13:42.294081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.294072    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:13:42.394865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.394834    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a738826-ed52-4e69-8623-04409ac092df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-047c00af-z7nq8\" (UID: \"4a738826-ed52-4e69-8623-04409ac092df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"
Nov 02 12:13:42.395076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.394905    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75wxb\" (UniqueName: \"kubernetes.io/projected/4a738826-ed52-4e69-8623-04409ac092df-kube-api-access-75wxb\") pod \"maas-api-revoke-sub-047c00af-z7nq8\" (UID: \"4a738826-ed52-4e69-8623-04409ac092df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"
Nov 02 12:13:42.395076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.394953    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a738826-ed52-4e69-8623-04409ac092df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-047c00af-z7nq8\" (UID: \"4a738826-ed52-4e69-8623-04409ac092df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"
Nov 02 12:13:42.395541 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.395519    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a738826-ed52-4e69-8623-04409ac092df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-047c00af-z7nq8\" (UID: \"4a738826-ed52-4e69-8623-04409ac092df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"
Nov 02 12:13:42.397340 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.397317    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a738826-ed52-4e69-8623-04409ac092df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-047c00af-z7nq8\" (UID: \"4a738826-ed52-4e69-8623-04409ac092df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"
Nov 02 12:13:42.465886 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.465808    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75wxb\" (UniqueName: \"kubernetes.io/projected/4a738826-ed52-4e69-8623-04409ac092df-kube-api-access-75wxb\") pod \"maas-api-revoke-sub-047c00af-z7nq8\" (UID: \"4a738826-ed52-4e69-8623-04409ac092df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"
Nov 02 12:13:42.504618 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.504589    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"
Nov 02 12:13:42.883291 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:42.883261    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"]
Nov 02 12:13:42.883407 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:13:42.883309    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a738826_ed52_4e69_8623_04409ac092df.slice/crio-e42da4bf1e662917a75ef372fca5b0f5b9f2e3b64907838995300a977c6eb94b WatchSource:0}: Error finding container e42da4bf1e662917a75ef372fca5b0f5b9f2e3b64907838995300a977c6eb94b: Status 404 returned error can't find the container with id e42da4bf1e662917a75ef372fca5b0f5b9f2e3b64907838995300a977c6eb94b
Nov 02 12:13:43.407029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:43.406966    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8" event={"ID":"4a738826-ed52-4e69-8623-04409ac092df","Type":"ContainerStarted","Data":"039cd0614daace0a6981d555275c791de78e5403f95c50038a80e22be975c4ca"}
Nov 02 12:13:43.407441 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:43.407055    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8" event={"ID":"4a738826-ed52-4e69-8623-04409ac092df","Type":"ContainerStarted","Data":"e42da4bf1e662917a75ef372fca5b0f5b9f2e3b64907838995300a977c6eb94b"}
Nov 02 12:13:43.525859 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:43.525803    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8" podStartSLOduration=1.525789655 podStartE2EDuration="1.525789655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:13:43.523647467 +0000 UTC m=+1052.087355879" watchObservedRunningTime="2026-11-02 12:13:43.525789655 +0000 UTC m=+1052.089498064"
Nov 02 12:13:44.415285 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:44.415248    4464 generic.go:356] "Generic (PLEG): container finished" podID="4a738826-ed52-4e69-8623-04409ac092df" containerID="039cd0614daace0a6981d555275c791de78e5403f95c50038a80e22be975c4ca" exitCode=0
Nov 02 12:13:44.415739 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:44.415333    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8" event={"ID":"4a738826-ed52-4e69-8623-04409ac092df","Type":"ContainerDied","Data":"039cd0614daace0a6981d555275c791de78e5403f95c50038a80e22be975c4ca"}
Nov 02 12:13:45.577628 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:45.577599    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"
Nov 02 12:13:45.630514 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:45.630487    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a738826-ed52-4e69-8623-04409ac092df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a738826-ed52-4e69-8623-04409ac092df-maas-api-openshift-ca\") pod \"4a738826-ed52-4e69-8623-04409ac092df\" (UID: \"4a738826-ed52-4e69-8623-04409ac092df\") "
Nov 02 12:13:45.630664 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:45.630650    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a738826-ed52-4e69-8623-04409ac092df-kube-api-access-75wxb\" (UniqueName: \"kubernetes.io/projected/4a738826-ed52-4e69-8623-04409ac092df-kube-api-access-75wxb\") pod \"4a738826-ed52-4e69-8623-04409ac092df\" (UID: \"4a738826-ed52-4e69-8623-04409ac092df\") "
Nov 02 12:13:45.630717 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:45.630683    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a738826-ed52-4e69-8623-04409ac092df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a738826-ed52-4e69-8623-04409ac092df-maas-api-xks-ca\") pod \"4a738826-ed52-4e69-8623-04409ac092df\" (UID: \"4a738826-ed52-4e69-8623-04409ac092df\") "
Nov 02 12:13:45.630899 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:45.630866    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a738826-ed52-4e69-8623-04409ac092df-maas-api-openshift-ca" pod "4a738826-ed52-4e69-8623-04409ac092df" (UID: "4a738826-ed52-4e69-8623-04409ac092df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:13:45.631277 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:45.631252    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a738826-ed52-4e69-8623-04409ac092df-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:13:45.632801 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:45.632778    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a738826-ed52-4e69-8623-04409ac092df-maas-api-xks-ca" pod "4a738826-ed52-4e69-8623-04409ac092df" (UID: "4a738826-ed52-4e69-8623-04409ac092df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:13:45.632801 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:45.632790    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a738826-ed52-4e69-8623-04409ac092df-kube-api-access-75wxb" pod "4a738826-ed52-4e69-8623-04409ac092df" (UID: "4a738826-ed52-4e69-8623-04409ac092df"). InnerVolumeSpecName "kube-api-access-75wxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:13:45.732008 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:45.731958    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75wxb\" (UniqueName: \"kubernetes.io/projected/4a738826-ed52-4e69-8623-04409ac092df-kube-api-access-75wxb\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:13:45.732008 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:45.732003    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a738826-ed52-4e69-8623-04409ac092df-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:13:46.428394 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:46.428355    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"
Nov 02 12:13:46.428394 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:46.428371    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8" event={"ID":"4a738826-ed52-4e69-8623-04409ac092df","Type":"ContainerDied","Data":"e42da4bf1e662917a75ef372fca5b0f5b9f2e3b64907838995300a977c6eb94b"}
Nov 02 12:13:46.428599 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:46.428418    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e42da4bf1e662917a75ef372fca5b0f5b9f2e3b64907838995300a977c6eb94b"
Nov 02 12:13:47.920379 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:47.920337    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"]
Nov 02 12:13:47.921327 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:47.921306    4464 cpu_manager.go:414] "containerMap: removing container" podUID="4a738826-ed52-4e69-8623-04409ac092df" containerName="revoke-keys"
Nov 02 12:13:47.921393 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:47.921332    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a738826-ed52-4e69-8623-04409ac092df" containerName="revoke-keys"
Nov 02 12:13:47.921479 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:47.921466    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a738826-ed52-4e69-8623-04409ac092df" containerName="revoke-keys"
Nov 02 12:13:47.927139 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:47.927118    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.033165 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.033121    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4b9210\"/\"kube-root-ca.crt\""
Nov 02 12:13:48.033354 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.033186    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4b9210\"/\"default-dockercfg-6254k\""
Nov 02 12:13:48.033354 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.033220    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4b9210\"/\"distinct-2-w3-4b9210-kserve-self-signed-certs\""
Nov 02 12:13:48.033354 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.033249    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4b9210\"/\"openshift-service-ca.crt\""
Nov 02 12:13:48.036786 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.036764    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"]
Nov 02 12:13:48.068750 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.068698    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-home\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.078378 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.078345    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-model-cache\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.078679 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.078657    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-tmp-dir\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.078958 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.078938    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-dshm\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.087998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.079240    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d829dab9-d274-485e-bef3-a29c5ddfff22-tls-certs\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.089280 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.089240    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rzp6\" (UniqueName: \"kubernetes.io/projected/d829dab9-d274-485e-bef3-a29c5ddfff22-kube-api-access-7rzp6\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.191120 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.191017    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-dshm\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.191298 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.191160    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d829dab9-d274-485e-bef3-a29c5ddfff22-tls-certs\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.191298 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.191193    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rzp6\" (UniqueName: \"kubernetes.io/projected/d829dab9-d274-485e-bef3-a29c5ddfff22-kube-api-access-7rzp6\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.191298 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.191227    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-home\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.191298 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.191271    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-model-cache\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.191510 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.191325    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-tmp-dir\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.191749 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.191724    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-tmp-dir\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.191894 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.191842    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-home\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.192113 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.192087    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-model-cache\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.193894 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.193870    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-dshm\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.194288 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.194262    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d829dab9-d274-485e-bef3-a29c5ddfff22-tls-certs\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.253079 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.253050    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rzp6\" (UniqueName: \"kubernetes.io/projected/d829dab9-d274-485e-bef3-a29c5ddfff22-kube-api-access-7rzp6\") pod \"distinct-2-w3-4b9210-kserve-768464c84f-pg4mv\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.539069 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.539026    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:48.959478 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:48.959447    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"]
Nov 02 12:13:49.467000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:49.466953    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv" event={"ID":"d829dab9-d274-485e-bef3-a29c5ddfff22","Type":"ContainerStarted","Data":"5c2fc959481138c73b34d36eabd1e1830ab8bd27770e1c87f0448a1e8ff15b5a"}
Nov 02 12:13:49.467167 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:49.467009    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv" event={"ID":"d829dab9-d274-485e-bef3-a29c5ddfff22","Type":"ContainerStarted","Data":"90ce8ede3f467618e179bf1507ac057f952a2ca9c7a2d803ca9c1507ae137af1"}
Nov 02 12:13:49.539283 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:49.539258    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:49.589066 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:49.589016    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv" podStartSLOduration=2.58900336 podStartE2EDuration="2.58900336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:13:49.5861168 +0000 UTC m=+1058.149825209" watchObservedRunningTime="2026-11-02 12:13:49.58900336 +0000 UTC m=+1058.152711777"
Nov 02 12:13:58.292214 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:13:58.292177    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:13:58.539488 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:58.539457    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:58.552554 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:58.552490    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:13:58.780151 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:58.780116    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:13:59.542130 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:13:59.542100    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:14:04.060706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:04.060672    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"]
Nov 02 12:14:04.111164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:04.111132    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6798fd5-vmxnt"]
Nov 02 12:14:05.581771 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:05.581727    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49886a1f-ed79-4a50-a7fc-cef24cfa5992" path="/var/lib/kubelet/pods/49886a1f-ed79-4a50-a7fc-cef24cfa5992/volumes"
Nov 02 12:14:15.333862 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:15.333830    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"]
Nov 02 12:14:15.339113 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:15.339095    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"
Nov 02 12:14:15.392707 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:15.392681    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:14:15.392888 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:15.392745    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:14:15.392888 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:15.392820    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:14:15.393004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:15.392911    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:14:15.395910 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:15.395884    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"]
Nov 02 12:14:15.429282 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:15.429254    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5bsk\" (UniqueName: \"kubernetes.io/projected/74f3ea0f-f725-4620-93ab-47fc867878ce-kube-api-access-v5bsk\") pod \"maas-api-revoke-sub-156b3cc1-pkvvg\" (UID: \"74f3ea0f-f725-4620-93ab-47fc867878ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"
Nov 02 12:14:15.429439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:15.429403    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f3ea0f-f725-4620-93ab-47fc867878ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-156b3cc1-pkvvg\" (UID: \"74f3ea0f-f725-4620-93ab-47fc867878ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"
Nov 02 12:14:15.429439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:15.429433    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f3ea0f-f725-4620-93ab-47fc867878ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-156b3cc1-pkvvg\" (UID: \"74f3ea0f-f725-4620-93ab-47fc867878ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"
Nov 02 12:14:15.531037 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:15.530999    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5bsk\" (UniqueName: \"kubernetes.io/projected/74f3ea0f-f725-4620-93ab-47fc867878ce-kube-api-access-v5bsk\") pod \"maas-api-revoke-sub-156b3cc1-pkvvg\" (UID: \"74f3ea0f-f725-4620-93ab-47fc867878ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"
Nov 02 12:14:15.531183 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:15.531142    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f3ea0f-f725-4620-93ab-47fc867878ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-156b3cc1-pkvvg\" (UID: \"74f3ea0f-f725-4620-93ab-47fc867878ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"
Nov 02 12:14:15.531183 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:15.531171    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f3ea0f-f725-4620-93ab-47fc867878ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-156b3cc1-pkvvg\" (UID: \"74f3ea0f-f725-4620-93ab-47fc867878ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"
Nov 02 12:14:15.531847 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:15.531825    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f3ea0f-f725-4620-93ab-47fc867878ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-156b3cc1-pkvvg\" (UID: \"74f3ea0f-f725-4620-93ab-47fc867878ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"
Nov 02 12:14:15.533502 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:15.533478    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f3ea0f-f725-4620-93ab-47fc867878ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-156b3cc1-pkvvg\" (UID: \"74f3ea0f-f725-4620-93ab-47fc867878ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"
Nov 02 12:14:15.593012 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:15.592917    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5bsk\" (UniqueName: \"kubernetes.io/projected/74f3ea0f-f725-4620-93ab-47fc867878ce-kube-api-access-v5bsk\") pod \"maas-api-revoke-sub-156b3cc1-pkvvg\" (UID: \"74f3ea0f-f725-4620-93ab-47fc867878ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"
Nov 02 12:14:15.653111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:15.653082    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"
Nov 02 12:14:16.033350 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:16.033317    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"]
Nov 02 12:14:16.034510 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:14:16.034404    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f3ea0f_f725_4620_93ab_47fc867878ce.slice/crio-30aa57d9a87e22cefdffc8f3475f46423ede7ba83556905c5a8df0422690967d WatchSource:0}: Error finding container 30aa57d9a87e22cefdffc8f3475f46423ede7ba83556905c5a8df0422690967d: Status 404 returned error can't find the container with id 30aa57d9a87e22cefdffc8f3475f46423ede7ba83556905c5a8df0422690967d
Nov 02 12:14:16.699558 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:16.699457    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg" event={"ID":"74f3ea0f-f725-4620-93ab-47fc867878ce","Type":"ContainerStarted","Data":"101421be2f46dbba9045b209493abd5ac2bc84e3e373843589a330d953f7d0e6"}
Nov 02 12:14:16.700196 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:16.700169    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg" event={"ID":"74f3ea0f-f725-4620-93ab-47fc867878ce","Type":"ContainerStarted","Data":"30aa57d9a87e22cefdffc8f3475f46423ede7ba83556905c5a8df0422690967d"}
Nov 02 12:14:17.707956 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:17.707924    4464 generic.go:356] "Generic (PLEG): container finished" podID="74f3ea0f-f725-4620-93ab-47fc867878ce" containerID="101421be2f46dbba9045b209493abd5ac2bc84e3e373843589a330d953f7d0e6" exitCode=0
Nov 02 12:14:17.708331 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:17.707968    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg" event={"ID":"74f3ea0f-f725-4620-93ab-47fc867878ce","Type":"ContainerDied","Data":"101421be2f46dbba9045b209493abd5ac2bc84e3e373843589a330d953f7d0e6"}
Nov 02 12:14:18.685496 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:18.685457    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"]
Nov 02 12:14:18.733615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:18.733581    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b6f3d8-vqnpr"]
Nov 02 12:14:18.856857 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:18.856830    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"
Nov 02 12:14:18.932400 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:18.932366    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f3ea0f-f725-4620-93ab-47fc867878ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f3ea0f-f725-4620-93ab-47fc867878ce-maas-api-openshift-ca\") pod \"74f3ea0f-f725-4620-93ab-47fc867878ce\" (UID: \"74f3ea0f-f725-4620-93ab-47fc867878ce\") "
Nov 02 12:14:18.932665 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:18.932508    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f3ea0f-f725-4620-93ab-47fc867878ce-kube-api-access-v5bsk\" (UniqueName: \"kubernetes.io/projected/74f3ea0f-f725-4620-93ab-47fc867878ce-kube-api-access-v5bsk\") pod \"74f3ea0f-f725-4620-93ab-47fc867878ce\" (UID: \"74f3ea0f-f725-4620-93ab-47fc867878ce\") "
Nov 02 12:14:18.932665 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:18.932589    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74f3ea0f-f725-4620-93ab-47fc867878ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f3ea0f-f725-4620-93ab-47fc867878ce-maas-api-xks-ca\") pod \"74f3ea0f-f725-4620-93ab-47fc867878ce\" (UID: \"74f3ea0f-f725-4620-93ab-47fc867878ce\") "
Nov 02 12:14:18.932807 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:18.932732    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f3ea0f-f725-4620-93ab-47fc867878ce-maas-api-openshift-ca" pod "74f3ea0f-f725-4620-93ab-47fc867878ce" (UID: "74f3ea0f-f725-4620-93ab-47fc867878ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:14:18.933075 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:18.933054    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f3ea0f-f725-4620-93ab-47fc867878ce-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:18.936087 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:18.936021    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f3ea0f-f725-4620-93ab-47fc867878ce-kube-api-access-v5bsk" pod "74f3ea0f-f725-4620-93ab-47fc867878ce" (UID: "74f3ea0f-f725-4620-93ab-47fc867878ce"). InnerVolumeSpecName "kube-api-access-v5bsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:14:18.936808 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:18.936787    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f3ea0f-f725-4620-93ab-47fc867878ce-maas-api-xks-ca" pod "74f3ea0f-f725-4620-93ab-47fc867878ce" (UID: "74f3ea0f-f725-4620-93ab-47fc867878ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:14:18.949695 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:18.949674    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"]
Nov 02 12:14:18.949892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:18.949871    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9" podUID="739b7556-be7f-4208-b953-4f82f7511f3e" containerName="istio-proxy" containerID="cri-o://4b5a4d1d84cb505cd351a8dab6c80d886b2330ea3de36739490c3a13933c0f63" gracePeriod=30
Nov 02 12:14:19.034456 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:19.034425    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5bsk\" (UniqueName: \"kubernetes.io/projected/74f3ea0f-f725-4620-93ab-47fc867878ce-kube-api-access-v5bsk\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:19.034456 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:19.034455    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f3ea0f-f725-4620-93ab-47fc867878ce-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:19.583367 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:19.583333    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9" path="/var/lib/kubelet/pods/ad1fc565-c18d-47bb-a6f2-e3fe68afd8a9/volumes"
Nov 02 12:14:19.726540 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:19.726500    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg" event={"ID":"74f3ea0f-f725-4620-93ab-47fc867878ce","Type":"ContainerDied","Data":"30aa57d9a87e22cefdffc8f3475f46423ede7ba83556905c5a8df0422690967d"}
Nov 02 12:14:19.726540 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:19.726526    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"
Nov 02 12:14:19.726751 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:19.726565    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30aa57d9a87e22cefdffc8f3475f46423ede7ba83556905c5a8df0422690967d"
Nov 02 12:14:24.208040 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.208015    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:14:24.330003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.329956    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-workload-certs\") pod \"739b7556-be7f-4208-b953-4f82f7511f3e\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") "
Nov 02 12:14:24.330194 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.330042    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-workload-socket\") pod \"739b7556-be7f-4208-b953-4f82f7511f3e\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") "
Nov 02 12:14:24.330194 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.330067    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/739b7556-be7f-4208-b953-4f82f7511f3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739b7556-be7f-4208-b953-4f82f7511f3e-istio-podinfo\") pod \"739b7556-be7f-4208-b953-4f82f7511f3e\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") "
Nov 02 12:14:24.330194 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.330115    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739b7556-be7f-4208-b953-4f82f7511f3e-kube-api-access-fl4pm\" (UniqueName: \"kubernetes.io/projected/739b7556-be7f-4208-b953-4f82f7511f3e-kube-api-access-fl4pm\") pod \"739b7556-be7f-4208-b953-4f82f7511f3e\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") "
Nov 02 12:14:24.330194 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.330142    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-istio-envoy\") pod \"739b7556-be7f-4208-b953-4f82f7511f3e\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") "
Nov 02 12:14:24.330399 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.330216    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-credential-socket\") pod \"739b7556-be7f-4208-b953-4f82f7511f3e\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") "
Nov 02 12:14:24.330399 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.330278    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-istio-data\") pod \"739b7556-be7f-4208-b953-4f82f7511f3e\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") "
Nov 02 12:14:24.330399 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.330324    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739b7556-be7f-4208-b953-4f82f7511f3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739b7556-be7f-4208-b953-4f82f7511f3e-istiod-ca-cert\") pod \"739b7556-be7f-4208-b953-4f82f7511f3e\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") "
Nov 02 12:14:24.330399 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.330331    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-workload-socket" pod "739b7556-be7f-4208-b953-4f82f7511f3e" (UID: "739b7556-be7f-4208-b953-4f82f7511f3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:14:24.330399 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.330377    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-workload-certs" pod "739b7556-be7f-4208-b953-4f82f7511f3e" (UID: "739b7556-be7f-4208-b953-4f82f7511f3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:14:24.330658 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.330431    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739b7556-be7f-4208-b953-4f82f7511f3e-istio-token\" (UniqueName: \"kubernetes.io/projected/739b7556-be7f-4208-b953-4f82f7511f3e-istio-token\") pod \"739b7556-be7f-4208-b953-4f82f7511f3e\" (UID: \"739b7556-be7f-4208-b953-4f82f7511f3e\") "
Nov 02 12:14:24.330658 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.330574    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-credential-socket" pod "739b7556-be7f-4208-b953-4f82f7511f3e" (UID: "739b7556-be7f-4208-b953-4f82f7511f3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:14:24.330658 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.330620    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-istio-data" pod "739b7556-be7f-4208-b953-4f82f7511f3e" (UID: "739b7556-be7f-4208-b953-4f82f7511f3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:14:24.330817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.330703    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739b7556-be7f-4208-b953-4f82f7511f3e-istiod-ca-cert" pod "739b7556-be7f-4208-b953-4f82f7511f3e" (UID: "739b7556-be7f-4208-b953-4f82f7511f3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:14:24.330994 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.330952    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:24.331081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.330994    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:24.331081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.331010    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:24.331081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.331023    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:24.331081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.331046    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/739b7556-be7f-4208-b953-4f82f7511f3e-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:24.332632 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.332604    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739b7556-be7f-4208-b953-4f82f7511f3e-kube-api-access-fl4pm" pod "739b7556-be7f-4208-b953-4f82f7511f3e" (UID: "739b7556-be7f-4208-b953-4f82f7511f3e"). InnerVolumeSpecName "kube-api-access-fl4pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:14:24.332723 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.332612    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/739b7556-be7f-4208-b953-4f82f7511f3e-istio-podinfo" pod "739b7556-be7f-4208-b953-4f82f7511f3e" (UID: "739b7556-be7f-4208-b953-4f82f7511f3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:14:24.332723 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.332636    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-istio-envoy" pod "739b7556-be7f-4208-b953-4f82f7511f3e" (UID: "739b7556-be7f-4208-b953-4f82f7511f3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:14:24.332878 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.332863    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739b7556-be7f-4208-b953-4f82f7511f3e-istio-token" pod "739b7556-be7f-4208-b953-4f82f7511f3e" (UID: "739b7556-be7f-4208-b953-4f82f7511f3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:14:24.433229 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.433195    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/739b7556-be7f-4208-b953-4f82f7511f3e-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:24.433229 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.433229    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/739b7556-be7f-4208-b953-4f82f7511f3e-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:24.433422 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.433244    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl4pm\" (UniqueName: \"kubernetes.io/projected/739b7556-be7f-4208-b953-4f82f7511f3e-kube-api-access-fl4pm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:24.433422 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.433260    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/739b7556-be7f-4208-b953-4f82f7511f3e-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:24.758961 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.758927    4464 generic.go:356] "Generic (PLEG): container finished" podID="739b7556-be7f-4208-b953-4f82f7511f3e" containerID="4b5a4d1d84cb505cd351a8dab6c80d886b2330ea3de36739490c3a13933c0f63" exitCode=0
Nov 02 12:14:24.759175 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.759008    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"
Nov 02 12:14:24.759175 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.759022    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9" event={"ID":"739b7556-be7f-4208-b953-4f82f7511f3e","Type":"ContainerDied","Data":"4b5a4d1d84cb505cd351a8dab6c80d886b2330ea3de36739490c3a13933c0f63"}
Nov 02 12:14:24.759175 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.759077    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9" event={"ID":"739b7556-be7f-4208-b953-4f82f7511f3e","Type":"ContainerDied","Data":"fc0a5f42059e4330acc47cda9d8e21647fb55742a5e840d14a75954970f79d8a"}
Nov 02 12:14:24.759175 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.759101    4464 scope.go:122] "RemoveContainer" containerID="4b5a4d1d84cb505cd351a8dab6c80d886b2330ea3de36739490c3a13933c0f63"
Nov 02 12:14:24.774052 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.773403    4464 scope.go:122] "RemoveContainer" containerID="a5eddea6d73d2504c8942f3c73e7ac577b68931e325a29cd44dedc1bb580f712"
Nov 02 12:14:24.783064 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.783047    4464 scope.go:122] "RemoveContainer" containerID="4b5a4d1d84cb505cd351a8dab6c80d886b2330ea3de36739490c3a13933c0f63"
Nov 02 12:14:24.783342 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:14:24.783323    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b5a4d1d84cb505cd351a8dab6c80d886b2330ea3de36739490c3a13933c0f63\": container with ID starting with 4b5a4d1d84cb505cd351a8dab6c80d886b2330ea3de36739490c3a13933c0f63 not found: ID does not exist" containerID="4b5a4d1d84cb505cd351a8dab6c80d886b2330ea3de36739490c3a13933c0f63"
Nov 02 12:14:24.783393 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.783352    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5a4d1d84cb505cd351a8dab6c80d886b2330ea3de36739490c3a13933c0f63"} err="failed to get container status \"4b5a4d1d84cb505cd351a8dab6c80d886b2330ea3de36739490c3a13933c0f63\": rpc error: code = NotFound desc = could not find container \"4b5a4d1d84cb505cd351a8dab6c80d886b2330ea3de36739490c3a13933c0f63\": container with ID starting with 4b5a4d1d84cb505cd351a8dab6c80d886b2330ea3de36739490c3a13933c0f63 not found: ID does not exist"
Nov 02 12:14:24.783393 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.783371    4464 scope.go:122] "RemoveContainer" containerID="a5eddea6d73d2504c8942f3c73e7ac577b68931e325a29cd44dedc1bb580f712"
Nov 02 12:14:24.783614 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:14:24.783599    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5eddea6d73d2504c8942f3c73e7ac577b68931e325a29cd44dedc1bb580f712\": container with ID starting with a5eddea6d73d2504c8942f3c73e7ac577b68931e325a29cd44dedc1bb580f712 not found: ID does not exist" containerID="a5eddea6d73d2504c8942f3c73e7ac577b68931e325a29cd44dedc1bb580f712"
Nov 02 12:14:24.783655 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.783621    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5eddea6d73d2504c8942f3c73e7ac577b68931e325a29cd44dedc1bb580f712"} err="failed to get container status \"a5eddea6d73d2504c8942f3c73e7ac577b68931e325a29cd44dedc1bb580f712\": rpc error: code = NotFound desc = could not find container \"a5eddea6d73d2504c8942f3c73e7ac577b68931e325a29cd44dedc1bb580f712\": container with ID starting with a5eddea6d73d2504c8942f3c73e7ac577b68931e325a29cd44dedc1bb580f712 not found: ID does not exist"
Nov 02 12:14:24.933587 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.933546    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"]
Nov 02 12:14:24.939662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:24.939635    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b6f3d8-openshift-default-848f7897df-zjxx9"]
Nov 02 12:14:25.039472 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.039386    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"]
Nov 02 12:14:25.040067 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.040048    4464 cpu_manager.go:414] "containerMap: removing container" podUID="739b7556-be7f-4208-b953-4f82f7511f3e" containerName="istio-proxy"
Nov 02 12:14:25.040167 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.040068    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739b7556-be7f-4208-b953-4f82f7511f3e" containerName="istio-proxy"
Nov 02 12:14:25.040167 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.040091    4464 cpu_manager.go:414] "containerMap: removing container" podUID="74f3ea0f-f725-4620-93ab-47fc867878ce" containerName="revoke-keys"
Nov 02 12:14:25.040167 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.040098    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f3ea0f-f725-4620-93ab-47fc867878ce" containerName="revoke-keys"
Nov 02 12:14:25.040325 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.040256    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="739b7556-be7f-4208-b953-4f82f7511f3e" containerName="istio-proxy"
Nov 02 12:14:25.040325 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.040270    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="739b7556-be7f-4208-b953-4f82f7511f3e" containerName="istio-proxy"
Nov 02 12:14:25.040325 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.040280    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f3ea0f-f725-4620-93ab-47fc867878ce" containerName="revoke-keys"
Nov 02 12:14:25.045378 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.045359    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.102098 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.102068    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7bd72e\"/\"kube-root-ca.crt\""
Nov 02 12:14:25.102286 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.102132    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7bd72e\"/\"openshift-service-ca.crt\""
Nov 02 12:14:25.102286 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.102206    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7bd72e\"/\"test-model-7bd72e-kserve-self-signed-certs\""
Nov 02 12:14:25.102406 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.102288    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7bd72e\"/\"default-dockercfg-w6c8q\""
Nov 02 12:14:25.104779 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.104755    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"]
Nov 02 12:14:25.141712 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.141685    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-home\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.141846 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.141725    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-tmp-dir\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.141846 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.141749    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-model-cache\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.141944 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.141923    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-dshm\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.142006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.141970    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkpps\" (UniqueName: \"kubernetes.io/projected/ff6c0679-4d25-499c-809d-4e22187c23ac-kube-api-access-vkpps\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.142109 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.142091    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff6c0679-4d25-499c-809d-4e22187c23ac-tls-certs\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.245016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.244580    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-dshm\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.245016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.244652    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkpps\" (UniqueName: \"kubernetes.io/projected/ff6c0679-4d25-499c-809d-4e22187c23ac-kube-api-access-vkpps\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.245016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.244734    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff6c0679-4d25-499c-809d-4e22187c23ac-tls-certs\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.245016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.244816    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-home\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.245016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.244854    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-tmp-dir\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.245016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.244887    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-model-cache\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.246307 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.246280    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-model-cache\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.246906 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.246882    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-home\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.247042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.247013    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-tmp-dir\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.248633 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.248606    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-dshm\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.249017 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.248995    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff6c0679-4d25-499c-809d-4e22187c23ac-tls-certs\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.311231 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.311159    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkpps\" (UniqueName: \"kubernetes.io/projected/ff6c0679-4d25-499c-809d-4e22187c23ac-kube-api-access-vkpps\") pod \"test-model-7bd72e-kserve-878dc8cbf-882jf\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") " pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.357030 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.357002    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:25.582659 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.582570    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="739b7556-be7f-4208-b953-4f82f7511f3e" path="/var/lib/kubelet/pods/739b7556-be7f-4208-b953-4f82f7511f3e/volumes"
Nov 02 12:14:25.732926 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.732897    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"]
Nov 02 12:14:25.733074 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:14:25.732947    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff6c0679_4d25_499c_809d_4e22187c23ac.slice/crio-005dbf177be9a39d9179caad3b21d96456acf3d90f58c695f636f9bf0b75aa48 WatchSource:0}: Error finding container 005dbf177be9a39d9179caad3b21d96456acf3d90f58c695f636f9bf0b75aa48: Status 404 returned error can't find the container with id 005dbf177be9a39d9179caad3b21d96456acf3d90f58c695f636f9bf0b75aa48
Nov 02 12:14:25.768706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:25.768684    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf" event={"ID":"ff6c0679-4d25-499c-809d-4e22187c23ac","Type":"ContainerStarted","Data":"005dbf177be9a39d9179caad3b21d96456acf3d90f58c695f636f9bf0b75aa48"}
Nov 02 12:14:26.777096 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:26.777060    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf" event={"ID":"ff6c0679-4d25-499c-809d-4e22187c23ac","Type":"ContainerStarted","Data":"2d2b3f52c904fa0cee2ec422c89a98fd092c138e77e0ca90d2504143ce9bdbdf"}
Nov 02 12:14:26.896872 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:26.896815    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf" podStartSLOduration=2.896801371 podStartE2EDuration="2.896801371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:14:26.895258787 +0000 UTC m=+1095.458967221" watchObservedRunningTime="2026-11-02 12:14:26.896801371 +0000 UTC m=+1095.460509780"
Nov 02 12:14:27.357737 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:27.357704    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:28.291949 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:14:28.291913    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:14:30.353423 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:30.353386    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:14:32.128580 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:32.128540    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"]
Nov 02 12:14:32.243108 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:32.243068    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"]
Nov 02 12:14:33.217375 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.217340    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"]
Nov 02 12:14:33.217942 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.217928    4464 cpu_manager.go:414] "containerMap: removing container" podUID="739b7556-be7f-4208-b953-4f82f7511f3e" containerName="istio-proxy"
Nov 02 12:14:33.218008 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.217943    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739b7556-be7f-4208-b953-4f82f7511f3e" containerName="istio-proxy"
Nov 02 12:14:33.221443 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.221423    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"
Nov 02 12:14:33.280628 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.280598    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"]
Nov 02 12:14:33.281295 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.281270    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:14:33.281469 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.281310    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:14:33.281580 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.281408    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:14:33.281731 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.281711    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:14:33.367057 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.367025    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-156e3027-nml6k\" (UID: \"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"
Nov 02 12:14:33.367057 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.367061    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-262kh\" (UniqueName: \"kubernetes.io/projected/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-kube-api-access-262kh\") pod \"maas-api-revoke-sub-156e3027-nml6k\" (UID: \"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"
Nov 02 12:14:33.367242 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.367153    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-156e3027-nml6k\" (UID: \"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"
Nov 02 12:14:33.468408 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.468380    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-156e3027-nml6k\" (UID: \"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"
Nov 02 12:14:33.468547 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.468418    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-262kh\" (UniqueName: \"kubernetes.io/projected/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-kube-api-access-262kh\") pod \"maas-api-revoke-sub-156e3027-nml6k\" (UID: \"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"
Nov 02 12:14:33.468547 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.468470    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-156e3027-nml6k\" (UID: \"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"
Nov 02 12:14:33.469221 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.469192    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-156e3027-nml6k\" (UID: \"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"
Nov 02 12:14:33.470911 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.470837    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-156e3027-nml6k\" (UID: \"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"
Nov 02 12:14:33.530757 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.530729    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-262kh\" (UniqueName: \"kubernetes.io/projected/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-kube-api-access-262kh\") pod \"maas-api-revoke-sub-156e3027-nml6k\" (UID: \"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"
Nov 02 12:14:33.535238 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.535214    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"
Nov 02 12:14:33.969249 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:33.967923    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"]
Nov 02 12:14:34.135104 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.135074    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"]
Nov 02 12:14:34.140332 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.140315    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"
Nov 02 12:14:34.197878 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.197854    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"]
Nov 02 12:14:34.209352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.209329    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xjcf\" (UniqueName: \"kubernetes.io/projected/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-kube-api-access-9xjcf\") pod \"maas-api-revoke-sub-460b9029-5c9j7\" (UID: \"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"
Nov 02 12:14:34.209456 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.209414    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-460b9029-5c9j7\" (UID: \"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"
Nov 02 12:14:34.209592 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.209568    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-460b9029-5c9j7\" (UID: \"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"
Nov 02 12:14:34.310444 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.310415    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-460b9029-5c9j7\" (UID: \"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"
Nov 02 12:14:34.310849 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.310495    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-460b9029-5c9j7\" (UID: \"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"
Nov 02 12:14:34.310849 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.310551    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xjcf\" (UniqueName: \"kubernetes.io/projected/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-kube-api-access-9xjcf\") pod \"maas-api-revoke-sub-460b9029-5c9j7\" (UID: \"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"
Nov 02 12:14:34.311180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.311157    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-460b9029-5c9j7\" (UID: \"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"
Nov 02 12:14:34.312621 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.312601    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-460b9029-5c9j7\" (UID: \"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"
Nov 02 12:14:34.372272 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.372244    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xjcf\" (UniqueName: \"kubernetes.io/projected/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-kube-api-access-9xjcf\") pod \"maas-api-revoke-sub-460b9029-5c9j7\" (UID: \"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"
Nov 02 12:14:34.454058 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.454034    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"
Nov 02 12:14:34.833219 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:14:34.833192    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db3ca68_49f4_4ab1_b95d_4f34168c0a5c.slice/crio-e11fbfec059ccc95a3d4ad944da8e125debecee62d3b2fae008aaa09c61db89b WatchSource:0}: Error finding container e11fbfec059ccc95a3d4ad944da8e125debecee62d3b2fae008aaa09c61db89b: Status 404 returned error can't find the container with id e11fbfec059ccc95a3d4ad944da8e125debecee62d3b2fae008aaa09c61db89b
Nov 02 12:14:34.833579 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.833555    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"]
Nov 02 12:14:34.856264 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.856235    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k" event={"ID":"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4","Type":"ContainerStarted","Data":"a0636e8fe4f2c58ffb52388ed50cc5dee6d41f84384ef7776828c0b6e40c316f"}
Nov 02 12:14:34.856355 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.856287    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k" event={"ID":"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4","Type":"ContainerStarted","Data":"e167db9824a18faf9cecc64bcccfd77d9b830ce227365f09a21252822da9f427"}
Nov 02 12:14:34.857490 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.857468    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7" event={"ID":"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c","Type":"ContainerStarted","Data":"e11fbfec059ccc95a3d4ad944da8e125debecee62d3b2fae008aaa09c61db89b"}
Nov 02 12:14:34.973354 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:34.973306    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k" podStartSLOduration=1.973291718 podStartE2EDuration="1.973291718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:14:34.972908706 +0000 UTC m=+1103.536617127" watchObservedRunningTime="2026-11-02 12:14:34.973291718 +0000 UTC m=+1103.537000127"
Nov 02 12:14:35.357948 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:35.357870    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:35.369968 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:35.369940    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:35.864719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:35.864690    4464 generic.go:356] "Generic (PLEG): container finished" podID="6db3ca68-49f4-4ab1-b95d-4f34168c0a5c" containerID="7104dadda3e3f3a43058179907dcdd6102412fb32286db2b7ab27f2bc64906f8" exitCode=0
Nov 02 12:14:35.864829 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:35.864774    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7" event={"ID":"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c","Type":"ContainerDied","Data":"7104dadda3e3f3a43058179907dcdd6102412fb32286db2b7ab27f2bc64906f8"}
Nov 02 12:14:35.866402 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:35.866375    4464 generic.go:356] "Generic (PLEG): container finished" podID="52936383-9095-430a-9f24-6b8a649ee7c8" containerID="d8803c291f106aa35b77c6d6fb85fe6179ece3839b2aa291890e362ae15b3f58" exitCode=0
Nov 02 12:14:35.866506 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:35.866424    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vbstg-xx28x" event={"ID":"52936383-9095-430a-9f24-6b8a649ee7c8","Type":"ContainerDied","Data":"d8803c291f106aa35b77c6d6fb85fe6179ece3839b2aa291890e362ae15b3f58"}
Nov 02 12:14:35.867948 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:35.867925    4464 generic.go:356] "Generic (PLEG): container finished" podID="ec1555fb-9336-4fb5-8dbf-11e4e95e23f4" containerID="a0636e8fe4f2c58ffb52388ed50cc5dee6d41f84384ef7776828c0b6e40c316f" exitCode=0
Nov 02 12:14:35.868053 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:35.867949    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k" event={"ID":"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4","Type":"ContainerDied","Data":"a0636e8fe4f2c58ffb52388ed50cc5dee6d41f84384ef7776828c0b6e40c316f"}
Nov 02 12:14:35.879500 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:35.879453    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:14:37.050271 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.050246    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vbstg-xx28x"
Nov 02 12:14:37.075534 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.075506    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"
Nov 02 12:14:37.078954 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.078932    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"
Nov 02 12:14:37.129714 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.129647    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx" podUID="4d832815-bc49-464b-808c-70a5116c2fed" containerName="payload-processing" containerID="cri-o://50e9586c153055ae1f10af17276af5e4411d49172effeba2517cb82dafc9e268" gracePeriod=25
Nov 02 12:14:37.149136 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.149115    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-maas-api-xks-ca\") pod \"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c\" (UID: \"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c\") "
Nov 02 12:14:37.149220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.149201    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-kube-api-access-262kh\" (UniqueName: \"kubernetes.io/projected/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-kube-api-access-262kh\") pod \"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4\" (UID: \"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4\") "
Nov 02 12:14:37.149265 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.149254    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52936383-9095-430a-9f24-6b8a649ee7c8-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52936383-9095-430a-9f24-6b8a649ee7c8-service-ca-bundle\") pod \"52936383-9095-430a-9f24-6b8a649ee7c8\" (UID: \"52936383-9095-430a-9f24-6b8a649ee7c8\") "
Nov 02 12:14:37.149355 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.149291    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-maas-api-openshift-ca\") pod \"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4\" (UID: \"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4\") "
Nov 02 12:14:37.149355 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.149327    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-maas-api-xks-ca\") pod \"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4\" (UID: \"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4\") "
Nov 02 12:14:37.149355 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.149348    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-kube-api-access-9xjcf\" (UniqueName: \"kubernetes.io/projected/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-kube-api-access-9xjcf\") pod \"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c\" (UID: \"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c\") "
Nov 02 12:14:37.149505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.149385    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52936383-9095-430a-9f24-6b8a649ee7c8-kube-api-access-6spj4\" (UniqueName: \"kubernetes.io/projected/52936383-9095-430a-9f24-6b8a649ee7c8-kube-api-access-6spj4\") pod \"52936383-9095-430a-9f24-6b8a649ee7c8\" (UID: \"52936383-9095-430a-9f24-6b8a649ee7c8\") "
Nov 02 12:14:37.149505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.149431    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52936383-9095-430a-9f24-6b8a649ee7c8-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/52936383-9095-430a-9f24-6b8a649ee7c8-archives-path\") pod \"52936383-9095-430a-9f24-6b8a649ee7c8\" (UID: \"52936383-9095-430a-9f24-6b8a649ee7c8\") "
Nov 02 12:14:37.149505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.149468    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52936383-9095-430a-9f24-6b8a649ee7c8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52936383-9095-430a-9f24-6b8a649ee7c8-trusted-ca-bundle\") pod \"52936383-9095-430a-9f24-6b8a649ee7c8\" (UID: \"52936383-9095-430a-9f24-6b8a649ee7c8\") "
Nov 02 12:14:37.149666 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.149504    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-maas-api-openshift-ca\") pod \"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c\" (UID: \"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c\") "
Nov 02 12:14:37.149666 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.149640    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-maas-api-openshift-ca" pod "ec1555fb-9336-4fb5-8dbf-11e4e95e23f4" (UID: "ec1555fb-9336-4fb5-8dbf-11e4e95e23f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:14:37.149767 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.149702    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52936383-9095-430a-9f24-6b8a649ee7c8-service-ca-bundle" pod "52936383-9095-430a-9f24-6b8a649ee7c8" (UID: "52936383-9095-430a-9f24-6b8a649ee7c8"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:14:37.150140 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.149969    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-maas-api-openshift-ca" pod "6db3ca68-49f4-4ab1-b95d-4f34168c0a5c" (UID: "6db3ca68-49f4-4ab1-b95d-4f34168c0a5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:14:37.150140 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.150027    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52936383-9095-430a-9f24-6b8a649ee7c8-archives-path" pod "52936383-9095-430a-9f24-6b8a649ee7c8" (UID: "52936383-9095-430a-9f24-6b8a649ee7c8"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:14:37.150140 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.150076    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:37.150140 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.150095    4464 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52936383-9095-430a-9f24-6b8a649ee7c8-service-ca-bundle\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:37.150140 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.150105    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:37.150397 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.150372    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52936383-9095-430a-9f24-6b8a649ee7c8-trusted-ca-bundle" pod "52936383-9095-430a-9f24-6b8a649ee7c8" (UID: "52936383-9095-430a-9f24-6b8a649ee7c8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:14:37.151955 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.151928    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52936383-9095-430a-9f24-6b8a649ee7c8-kube-api-access-6spj4" pod "52936383-9095-430a-9f24-6b8a649ee7c8" (UID: "52936383-9095-430a-9f24-6b8a649ee7c8"). InnerVolumeSpecName "kube-api-access-6spj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:14:37.151955 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.151938    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-kube-api-access-262kh" pod "ec1555fb-9336-4fb5-8dbf-11e4e95e23f4" (UID: "ec1555fb-9336-4fb5-8dbf-11e4e95e23f4"). InnerVolumeSpecName "kube-api-access-262kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:14:37.152123 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.151969    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-maas-api-xks-ca" pod "6db3ca68-49f4-4ab1-b95d-4f34168c0a5c" (UID: "6db3ca68-49f4-4ab1-b95d-4f34168c0a5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:14:37.152259 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.152239    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-maas-api-xks-ca" pod "ec1555fb-9336-4fb5-8dbf-11e4e95e23f4" (UID: "ec1555fb-9336-4fb5-8dbf-11e4e95e23f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:14:37.152259 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.152248    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-kube-api-access-9xjcf" pod "6db3ca68-49f4-4ab1-b95d-4f34168c0a5c" (UID: "6db3ca68-49f4-4ab1-b95d-4f34168c0a5c"). InnerVolumeSpecName "kube-api-access-9xjcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:14:37.243965 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.243912    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h" podUID="e0675eaf-e7c8-4516-bc84-9796fcdd505d" containerName="payload-pre-processing" containerID="cri-o://a3f17dfab3b721d5ef97caa042d3643241e5889df564c6498f1989ecc1835bc9" gracePeriod=25
Nov 02 12:14:37.251317 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.251278    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6spj4\" (UniqueName: \"kubernetes.io/projected/52936383-9095-430a-9f24-6b8a649ee7c8-kube-api-access-6spj4\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:37.251317 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.251306    4464 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/52936383-9095-430a-9f24-6b8a649ee7c8-archives-path\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:37.251317 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.251318    4464 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52936383-9095-430a-9f24-6b8a649ee7c8-trusted-ca-bundle\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:37.251530 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.251331    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:37.251530 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.251343    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-262kh\" (UniqueName: \"kubernetes.io/projected/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-kube-api-access-262kh\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:37.251530 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.251352    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:37.251530 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.251361    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xjcf\" (UniqueName: \"kubernetes.io/projected/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c-kube-api-access-9xjcf\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:37.397935 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.397910    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"
Nov 02 12:14:37.529314 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.529283    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"
Nov 02 12:14:37.555402 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.555370    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d832815-bc49-464b-808c-70a5116c2fed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d832815-bc49-464b-808c-70a5116c2fed-plugins-config-volume\") pod \"4d832815-bc49-464b-808c-70a5116c2fed\" (UID: \"4d832815-bc49-464b-808c-70a5116c2fed\") "
Nov 02 12:14:37.555532 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.555464    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d832815-bc49-464b-808c-70a5116c2fed-kube-api-access-mxfgk\" (UniqueName: \"kubernetes.io/projected/4d832815-bc49-464b-808c-70a5116c2fed-kube-api-access-mxfgk\") pod \"4d832815-bc49-464b-808c-70a5116c2fed\" (UID: \"4d832815-bc49-464b-808c-70a5116c2fed\") "
Nov 02 12:14:37.555817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.555784    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d832815-bc49-464b-808c-70a5116c2fed-plugins-config-volume" pod "4d832815-bc49-464b-808c-70a5116c2fed" (UID: "4d832815-bc49-464b-808c-70a5116c2fed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:14:37.556008 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.555988    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d832815-bc49-464b-808c-70a5116c2fed-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:37.557485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.557463    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d832815-bc49-464b-808c-70a5116c2fed-kube-api-access-mxfgk" pod "4d832815-bc49-464b-808c-70a5116c2fed" (UID: "4d832815-bc49-464b-808c-70a5116c2fed"). InnerVolumeSpecName "kube-api-access-mxfgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:14:37.656909 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.656843    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0675eaf-e7c8-4516-bc84-9796fcdd505d-kube-api-access-hcvp6\" (UniqueName: \"kubernetes.io/projected/e0675eaf-e7c8-4516-bc84-9796fcdd505d-kube-api-access-hcvp6\") pod \"e0675eaf-e7c8-4516-bc84-9796fcdd505d\" (UID: \"e0675eaf-e7c8-4516-bc84-9796fcdd505d\") "
Nov 02 12:14:37.657076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.657035    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0675eaf-e7c8-4516-bc84-9796fcdd505d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0675eaf-e7c8-4516-bc84-9796fcdd505d-plugins-config-volume\") pod \"e0675eaf-e7c8-4516-bc84-9796fcdd505d\" (UID: \"e0675eaf-e7c8-4516-bc84-9796fcdd505d\") "
Nov 02 12:14:37.657437 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.657416    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxfgk\" (UniqueName: \"kubernetes.io/projected/4d832815-bc49-464b-808c-70a5116c2fed-kube-api-access-mxfgk\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:37.657437 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.657427    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0675eaf-e7c8-4516-bc84-9796fcdd505d-plugins-config-volume" pod "e0675eaf-e7c8-4516-bc84-9796fcdd505d" (UID: "e0675eaf-e7c8-4516-bc84-9796fcdd505d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:14:37.659554 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.659525    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0675eaf-e7c8-4516-bc84-9796fcdd505d-kube-api-access-hcvp6" pod "e0675eaf-e7c8-4516-bc84-9796fcdd505d" (UID: "e0675eaf-e7c8-4516-bc84-9796fcdd505d"). InnerVolumeSpecName "kube-api-access-hcvp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:14:37.758319 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.758290    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcvp6\" (UniqueName: \"kubernetes.io/projected/e0675eaf-e7c8-4516-bc84-9796fcdd505d-kube-api-access-hcvp6\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:37.758319 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.758319    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0675eaf-e7c8-4516-bc84-9796fcdd505d-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:37.884848 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.884815    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vbstg-xx28x"
Nov 02 12:14:37.884848 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.884828    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vbstg-xx28x" event={"ID":"52936383-9095-430a-9f24-6b8a649ee7c8","Type":"ContainerDied","Data":"9207de42c8392fbaac63daf5c1e194ab1f73e526045bf1ac7fe1b55f45d12095"}
Nov 02 12:14:37.885119 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.884892    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9207de42c8392fbaac63daf5c1e194ab1f73e526045bf1ac7fe1b55f45d12095"
Nov 02 12:14:37.886485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.886460    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"
Nov 02 12:14:37.886609 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.886574    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k" event={"ID":"ec1555fb-9336-4fb5-8dbf-11e4e95e23f4","Type":"ContainerDied","Data":"e167db9824a18faf9cecc64bcccfd77d9b830ce227365f09a21252822da9f427"}
Nov 02 12:14:37.886722 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.886629    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e167db9824a18faf9cecc64bcccfd77d9b830ce227365f09a21252822da9f427"
Nov 02 12:14:37.888405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.888380    4464 generic.go:356] "Generic (PLEG): container finished" podID="e0675eaf-e7c8-4516-bc84-9796fcdd505d" containerID="a3f17dfab3b721d5ef97caa042d3643241e5889df564c6498f1989ecc1835bc9" exitCode=0
Nov 02 12:14:37.888505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.888415    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"
Nov 02 12:14:37.888505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.888474    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h" event={"ID":"e0675eaf-e7c8-4516-bc84-9796fcdd505d","Type":"ContainerDied","Data":"a3f17dfab3b721d5ef97caa042d3643241e5889df564c6498f1989ecc1835bc9"}
Nov 02 12:14:37.888958 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.888914    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h" event={"ID":"e0675eaf-e7c8-4516-bc84-9796fcdd505d","Type":"ContainerDied","Data":"f2c673e1686b0a13ea6025d21ca86ea2832c28f519cd7018961c8f93cb40debd"}
Nov 02 12:14:37.889083 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.888963    4464 scope.go:122] "RemoveContainer" containerID="a3f17dfab3b721d5ef97caa042d3643241e5889df564c6498f1989ecc1835bc9"
Nov 02 12:14:37.890646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.890339    4464 generic.go:356] "Generic (PLEG): container finished" podID="4d832815-bc49-464b-808c-70a5116c2fed" containerID="50e9586c153055ae1f10af17276af5e4411d49172effeba2517cb82dafc9e268" exitCode=0
Nov 02 12:14:37.890646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.890432    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx" event={"ID":"4d832815-bc49-464b-808c-70a5116c2fed","Type":"ContainerDied","Data":"50e9586c153055ae1f10af17276af5e4411d49172effeba2517cb82dafc9e268"}
Nov 02 12:14:37.890646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.890488    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx" event={"ID":"4d832815-bc49-464b-808c-70a5116c2fed","Type":"ContainerDied","Data":"b868bbc10d80a5d76d577af98d0fc834d53adac9193408bb0f123803a87c4716"}
Nov 02 12:14:37.890646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.890581    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"
Nov 02 12:14:37.894346 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.894315    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7" event={"ID":"6db3ca68-49f4-4ab1-b95d-4f34168c0a5c","Type":"ContainerDied","Data":"e11fbfec059ccc95a3d4ad944da8e125debecee62d3b2fae008aaa09c61db89b"}
Nov 02 12:14:37.894453 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.894360    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e11fbfec059ccc95a3d4ad944da8e125debecee62d3b2fae008aaa09c61db89b"
Nov 02 12:14:37.894453 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.894423    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"
Nov 02 12:14:37.900939 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.900919    4464 scope.go:122] "RemoveContainer" containerID="a3f17dfab3b721d5ef97caa042d3643241e5889df564c6498f1989ecc1835bc9"
Nov 02 12:14:37.901296 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:14:37.901278    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f17dfab3b721d5ef97caa042d3643241e5889df564c6498f1989ecc1835bc9\": container with ID starting with a3f17dfab3b721d5ef97caa042d3643241e5889df564c6498f1989ecc1835bc9 not found: ID does not exist" containerID="a3f17dfab3b721d5ef97caa042d3643241e5889df564c6498f1989ecc1835bc9"
Nov 02 12:14:37.901382 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.901301    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f17dfab3b721d5ef97caa042d3643241e5889df564c6498f1989ecc1835bc9"} err="failed to get container status \"a3f17dfab3b721d5ef97caa042d3643241e5889df564c6498f1989ecc1835bc9\": rpc error: code = NotFound desc = could not find container \"a3f17dfab3b721d5ef97caa042d3643241e5889df564c6498f1989ecc1835bc9\": container with ID starting with a3f17dfab3b721d5ef97caa042d3643241e5889df564c6498f1989ecc1835bc9 not found: ID does not exist"
Nov 02 12:14:37.901382 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.901319    4464 scope.go:122] "RemoveContainer" containerID="50e9586c153055ae1f10af17276af5e4411d49172effeba2517cb82dafc9e268"
Nov 02 12:14:37.914681 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.913703    4464 scope.go:122] "RemoveContainer" containerID="50e9586c153055ae1f10af17276af5e4411d49172effeba2517cb82dafc9e268"
Nov 02 12:14:37.914681 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:14:37.914037    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e9586c153055ae1f10af17276af5e4411d49172effeba2517cb82dafc9e268\": container with ID starting with 50e9586c153055ae1f10af17276af5e4411d49172effeba2517cb82dafc9e268 not found: ID does not exist" containerID="50e9586c153055ae1f10af17276af5e4411d49172effeba2517cb82dafc9e268"
Nov 02 12:14:37.914681 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:37.914069    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e9586c153055ae1f10af17276af5e4411d49172effeba2517cb82dafc9e268"} err="failed to get container status \"50e9586c153055ae1f10af17276af5e4411d49172effeba2517cb82dafc9e268\": rpc error: code = NotFound desc = could not find container \"50e9586c153055ae1f10af17276af5e4411d49172effeba2517cb82dafc9e268\": container with ID starting with 50e9586c153055ae1f10af17276af5e4411d49172effeba2517cb82dafc9e268 not found: ID does not exist"
Nov 02 12:14:38.111396 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.111359    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"]
Nov 02 12:14:38.114809 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.114786    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-966817-84c64b8595-7lkkx"]
Nov 02 12:14:38.230341 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.230314    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"]
Nov 02 12:14:38.230890 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.230877    4464 cpu_manager.go:414] "containerMap: removing container" podUID="52936383-9095-430a-9f24-6b8a649ee7c8" containerName="insights-gathering"
Nov 02 12:14:38.230950 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.230892    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52936383-9095-430a-9f24-6b8a649ee7c8" containerName="insights-gathering"
Nov 02 12:14:38.230950 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.230915    4464 cpu_manager.go:414] "containerMap: removing container" podUID="ec1555fb-9336-4fb5-8dbf-11e4e95e23f4" containerName="revoke-keys"
Nov 02 12:14:38.230950 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.230920    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1555fb-9336-4fb5-8dbf-11e4e95e23f4" containerName="revoke-keys"
Nov 02 12:14:38.230950 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.230929    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6db3ca68-49f4-4ab1-b95d-4f34168c0a5c" containerName="revoke-keys"
Nov 02 12:14:38.230950 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.230934    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db3ca68-49f4-4ab1-b95d-4f34168c0a5c" containerName="revoke-keys"
Nov 02 12:14:38.231171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.230958    4464 cpu_manager.go:414] "containerMap: removing container" podUID="4d832815-bc49-464b-808c-70a5116c2fed" containerName="payload-processing"
Nov 02 12:14:38.231171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.230963    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d832815-bc49-464b-808c-70a5116c2fed" containerName="payload-processing"
Nov 02 12:14:38.231171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.230970    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e0675eaf-e7c8-4516-bc84-9796fcdd505d" containerName="payload-pre-processing"
Nov 02 12:14:38.231171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.230989    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0675eaf-e7c8-4516-bc84-9796fcdd505d" containerName="payload-pre-processing"
Nov 02 12:14:38.231171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.231090    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="52936383-9095-430a-9f24-6b8a649ee7c8" containerName="insights-gathering"
Nov 02 12:14:38.231171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.231102    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1555fb-9336-4fb5-8dbf-11e4e95e23f4" containerName="revoke-keys"
Nov 02 12:14:38.231171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.231111    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db3ca68-49f4-4ab1-b95d-4f34168c0a5c" containerName="revoke-keys"
Nov 02 12:14:38.231171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.231120    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d832815-bc49-464b-808c-70a5116c2fed" containerName="payload-processing"
Nov 02 12:14:38.231171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.231125    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0675eaf-e7c8-4516-bc84-9796fcdd505d" containerName="payload-pre-processing"
Nov 02 12:14:38.236101 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.236081    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.236235 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.236215    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"]
Nov 02 12:14:38.241285 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.241258    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-966817-55764c99b7-dgp6h"]
Nov 02 12:14:38.311757 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.311720    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"]
Nov 02 12:14:38.312178 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.312151    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-036f88\"/\"openshift-service-ca.crt\""
Nov 02 12:14:38.312441 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.312421    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-036f88\"/\"kube-root-ca.crt\""
Nov 02 12:14:38.312565 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.312550    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-036f88\"/\"test-model-036f88-kserve-self-signed-certs\""
Nov 02 12:14:38.312681 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.312669    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-036f88\"/\"default-dockercfg-k46gc\""
Nov 02 12:14:38.385622 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.385590    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2njj\" (UniqueName: \"kubernetes.io/projected/df859e10-37e4-4d2e-a1d5-931adacbaaff-kube-api-access-g2njj\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.385775 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.385657    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-dshm\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.385775 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.385768    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-home\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.385849 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.385783    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-tmp-dir\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.385849 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.385809    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-model-cache\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.385849 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.385839    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df859e10-37e4-4d2e-a1d5-931adacbaaff-tls-certs\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.487569 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.487470    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-home\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.487569 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.487510    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-tmp-dir\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.487784 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.487647    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-model-cache\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.487784 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.487715    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df859e10-37e4-4d2e-a1d5-931adacbaaff-tls-certs\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.487906 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.487801    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2njj\" (UniqueName: \"kubernetes.io/projected/df859e10-37e4-4d2e-a1d5-931adacbaaff-kube-api-access-g2njj\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.487906 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.487880    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-dshm\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.488056 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.487925    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-home\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.488056 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.488013    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-tmp-dir\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.488527 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.488502    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-model-cache\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.497470 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.497444    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-dshm\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.497581 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.497512    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df859e10-37e4-4d2e-a1d5-931adacbaaff-tls-certs\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.550170 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.550140    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2njj\" (UniqueName: \"kubernetes.io/projected/df859e10-37e4-4d2e-a1d5-931adacbaaff-kube-api-access-g2njj\") pod \"test-model-036f88-kserve-766b784d9c-fcw72\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") " pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:38.853049 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:38.852485    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:39.286043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:39.286013    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"]
Nov 02 12:14:39.286403 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:14:39.285968    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf859e10_37e4_4d2e_a1d5_931adacbaaff.slice/crio-0c129eafe51b6e5793e3ee5d0a6b92ef3fafe65cebcb97f6b870563fa8d70dad WatchSource:0}: Error finding container 0c129eafe51b6e5793e3ee5d0a6b92ef3fafe65cebcb97f6b870563fa8d70dad: Status 404 returned error can't find the container with id 0c129eafe51b6e5793e3ee5d0a6b92ef3fafe65cebcb97f6b870563fa8d70dad
Nov 02 12:14:39.582387 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:39.582312    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d832815-bc49-464b-808c-70a5116c2fed" path="/var/lib/kubelet/pods/4d832815-bc49-464b-808c-70a5116c2fed/volumes"
Nov 02 12:14:39.582756 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:39.582742    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0675eaf-e7c8-4516-bc84-9796fcdd505d" path="/var/lib/kubelet/pods/e0675eaf-e7c8-4516-bc84-9796fcdd505d/volumes"
Nov 02 12:14:39.934854 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:39.934771    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72" event={"ID":"df859e10-37e4-4d2e-a1d5-931adacbaaff","Type":"ContainerStarted","Data":"c56c6bd30b55f719bab89f1de8076bdec8dcfe67936519d1d3abc20fc95825fd"}
Nov 02 12:14:39.934854 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:39.934816    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72" event={"ID":"df859e10-37e4-4d2e-a1d5-931adacbaaff","Type":"ContainerStarted","Data":"0c129eafe51b6e5793e3ee5d0a6b92ef3fafe65cebcb97f6b870563fa8d70dad"}
Nov 02 12:14:40.055448 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:40.055397    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72" podStartSLOduration=2.055385346 podStartE2EDuration="2.055385346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:14:40.054108585 +0000 UTC m=+1108.617816993" watchObservedRunningTime="2026-11-02 12:14:40.055385346 +0000 UTC m=+1108.619093761"
Nov 02 12:14:40.853557 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:40.853519    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:44.531563 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.531522    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"]
Nov 02 12:14:44.536838 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.536812    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.601953 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.601921    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"]
Nov 02 12:14:44.602216 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.602183    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5" podUID="ad3f68ba-d0bc-4ae1-b9c1-88850db07172" containerName="istio-proxy" containerID="cri-o://6f159818951a10416cd26ada0a95ab9d266f46f834f115b6e2a76ad97a966159" gracePeriod=30
Nov 02 12:14:44.642468 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.642436    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.642647 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.642506    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.642647 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.642545    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.642647 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.642572    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.642647 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.642605    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q5l2\" (UniqueName: \"kubernetes.io/projected/cad06365-a977-4f66-94eb-2511cc1d02f0-kube-api-access-2q5l2\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.642647 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.642638    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad06365-a977-4f66-94eb-2511cc1d02f0-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.642940 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.642684    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.642940 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.642814    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.642940 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.642857    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.650624 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.650591    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"]
Nov 02 12:14:44.751046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.749737    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.751046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.749803    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.751046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.749905    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.751046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.749957    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.751046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.750014    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.751046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.750043    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.751046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.750070    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5l2\" (UniqueName: \"kubernetes.io/projected/cad06365-a977-4f66-94eb-2511cc1d02f0-kube-api-access-2q5l2\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.751046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.750108    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad06365-a977-4f66-94eb-2511cc1d02f0-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.751046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.750155    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.751046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.750639    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.761572 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.754471    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.774009 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.769061    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad06365-a977-4f66-94eb-2511cc1d02f0-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.774009 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.769329    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.783038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.776375    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.783038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.781756    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.794183 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.794155    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.827749 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.824719    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.831767 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.831737    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q5l2\" (UniqueName: \"kubernetes.io/projected/cad06365-a977-4f66-94eb-2511cc1d02f0-kube-api-access-2q5l2\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:44.850634 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:44.850607    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:45.286274 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:45.286249    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"]
Nov 02 12:14:45.286538 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:14:45.286517    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad06365_a977_4f66_94eb_2511cc1d02f0.slice/crio-eea5e7f129c9c09f0766b81f48bff3b83ba9148010ed1b25af615c1cb0a30694 WatchSource:0}: Error finding container eea5e7f129c9c09f0766b81f48bff3b83ba9148010ed1b25af615c1cb0a30694: Status 404 returned error can't find the container with id eea5e7f129c9c09f0766b81f48bff3b83ba9148010ed1b25af615c1cb0a30694
Nov 02 12:14:45.288879 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:45.288846    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:14:45.288968 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:45.288927    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:14:45.289057 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:45.288986    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:14:45.992541 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:45.992503    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9" event={"ID":"cad06365-a977-4f66-94eb-2511cc1d02f0","Type":"ContainerStarted","Data":"0922e9cf37b9b2fff62d9a673d3e8d456fb5de6c330e8826209f3bd53fc4c443"}
Nov 02 12:14:45.992992 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:45.992563    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9" event={"ID":"cad06365-a977-4f66-94eb-2511cc1d02f0","Type":"ContainerStarted","Data":"eea5e7f129c9c09f0766b81f48bff3b83ba9148010ed1b25af615c1cb0a30694"}
Nov 02 12:14:46.118472 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:46.118411    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9" podStartSLOduration=2.118390941 podStartE2EDuration="2.118390941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:14:46.116272075 +0000 UTC m=+1114.679980498" watchObservedRunningTime="2026-11-02 12:14:46.118390941 +0000 UTC m=+1114.682099352"
Nov 02 12:14:46.851614 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:46.851577    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:46.853249 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:46.853226    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 12:14:46.853361 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:46.853285    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9" podUID="cad06365-a977-4f66-94eb-2511cc1d02f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 12:14:47.103038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.102954    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"]
Nov 02 12:14:47.107457 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.107436    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.131699 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.131669    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"]
Nov 02 12:14:47.131922 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.131899    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn" podUID="4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" containerName="istio-proxy" containerID="cri-o://6bf1c07240323add86bb44f0127a2f664f57b5a2afdc5eacbe75cbf2154e5aa3" gracePeriod=30
Nov 02 12:14:47.190099 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.176019    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-6ac03f-openshift-default-dockercfg-s52rh\""
Nov 02 12:14:47.203909 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.203875    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"]
Nov 02 12:14:47.329996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.319322    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istiod-ca-cert\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.329996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.319424    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rprb5\" (UniqueName: \"kubernetes.io/projected/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-kube-api-access-rprb5\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.329996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.319480    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-workload-certs\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.329996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.319554    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-credential-socket\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.329996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.319584    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-data\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.329996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.319622    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-token\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.329996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.319665    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-envoy\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.329996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.319706    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-podinfo\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.329996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.319768    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-workload-socket\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.443044 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.428940    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rprb5\" (UniqueName: \"kubernetes.io/projected/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-kube-api-access-rprb5\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.443044 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.429043    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-workload-certs\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.443044 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.429133    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-credential-socket\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.443044 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.429165    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-data\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.443044 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.429208    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-token\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.443044 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.429259    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-envoy\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.443044 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.429304    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-podinfo\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.443044 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.429373    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-workload-socket\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.443044 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.429413    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istiod-ca-cert\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.443044 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.430308    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istiod-ca-cert\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.443044 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.434014    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-credential-socket\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.443044 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.434967    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-workload-certs\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.443802 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.443777    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-workload-socket\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.444061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.444038    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-data\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.483428 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.483254    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-envoy\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.484863 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.484838    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-podinfo\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.496801 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.496775    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rprb5\" (UniqueName: \"kubernetes.io/projected/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-kube-api-access-rprb5\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.506050 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.506023    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-token\") pod \"e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") " pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.725283 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.725256    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:47.851691 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.851654    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 12:14:47.851847 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:47.851715    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9" podUID="cad06365-a977-4f66-94eb-2511cc1d02f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 12:14:48.189674 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:48.189623    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"]
Nov 02 12:14:48.191202 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:48.191168    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:14:48.191317 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:48.191248    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:14:48.191317 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:48.191293    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:14:48.850901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:48.850870    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 12:14:48.851088 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:48.850928    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9" podUID="cad06365-a977-4f66-94eb-2511cc1d02f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 12:14:48.852656 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:48.852633    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:48.892293 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:48.892263    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:49.019359 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:49.019323    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2" event={"ID":"90e7a43f-8da1-4a73-ab5e-f5edd3573d80","Type":"ContainerStarted","Data":"1ff1b5d3ae12317a255583ef40209d3a74e3519e9a2e859154157afb75f9c2b8"}
Nov 02 12:14:49.019530 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:49.019401    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2" event={"ID":"90e7a43f-8da1-4a73-ab5e-f5edd3573d80","Type":"ContainerStarted","Data":"345cf07c6ff7615a2851b6928d75288750cf56baafc063feb9c3b1bde198aa04"}
Nov 02 12:14:49.031220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:49.031191    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:14:49.152392 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:49.152297    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2" podStartSLOduration=2.152283308 podStartE2EDuration="2.152283308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:14:49.149242953 +0000 UTC m=+1117.712951363" watchObservedRunningTime="2026-11-02 12:14:49.152283308 +0000 UTC m=+1117.715991716"
Nov 02 12:14:49.726300 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:49.726268    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:49.732309 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:49.732088    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:49.851012 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:49.850951    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 12:14:49.851163 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:49.851079    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9" podUID="cad06365-a977-4f66-94eb-2511cc1d02f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 12:14:49.920713 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:49.920689    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:14:50.029555 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.029523    4464 generic.go:356] "Generic (PLEG): container finished" podID="ad3f68ba-d0bc-4ae1-b9c1-88850db07172" containerID="6f159818951a10416cd26ada0a95ab9d266f46f834f115b6e2a76ad97a966159" exitCode=0
Nov 02 12:14:50.029725 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.029593    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"
Nov 02 12:14:50.029725 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.029600    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5" event={"ID":"ad3f68ba-d0bc-4ae1-b9c1-88850db07172","Type":"ContainerDied","Data":"6f159818951a10416cd26ada0a95ab9d266f46f834f115b6e2a76ad97a966159"}
Nov 02 12:14:50.029725 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.029681    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5" event={"ID":"ad3f68ba-d0bc-4ae1-b9c1-88850db07172","Type":"ContainerDied","Data":"1abaad349b6bec4b9eb3988cab2b8a2d2cd420e59f0bbfdc5f8b999047ba3786"}
Nov 02 12:14:50.029725 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.029710    4464 scope.go:122] "RemoveContainer" containerID="6f159818951a10416cd26ada0a95ab9d266f46f834f115b6e2a76ad97a966159"
Nov 02 12:14:50.030214 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.030192    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:50.031127 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.031108    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:14:50.039809 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.039789    4464 scope.go:122] "RemoveContainer" containerID="6dba20abb675323d063d4d97e321c1a3a50116afbd189fd3fe03005338162c9d"
Nov 02 12:14:50.048859 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.048840    4464 scope.go:122] "RemoveContainer" containerID="6f159818951a10416cd26ada0a95ab9d266f46f834f115b6e2a76ad97a966159"
Nov 02 12:14:50.049140 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:14:50.049121    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f159818951a10416cd26ada0a95ab9d266f46f834f115b6e2a76ad97a966159\": container with ID starting with 6f159818951a10416cd26ada0a95ab9d266f46f834f115b6e2a76ad97a966159 not found: ID does not exist" containerID="6f159818951a10416cd26ada0a95ab9d266f46f834f115b6e2a76ad97a966159"
Nov 02 12:14:50.049191 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.049147    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f159818951a10416cd26ada0a95ab9d266f46f834f115b6e2a76ad97a966159"} err="failed to get container status \"6f159818951a10416cd26ada0a95ab9d266f46f834f115b6e2a76ad97a966159\": rpc error: code = NotFound desc = could not find container \"6f159818951a10416cd26ada0a95ab9d266f46f834f115b6e2a76ad97a966159\": container with ID starting with 6f159818951a10416cd26ada0a95ab9d266f46f834f115b6e2a76ad97a966159 not found: ID does not exist"
Nov 02 12:14:50.049191 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.049163    4464 scope.go:122] "RemoveContainer" containerID="6dba20abb675323d063d4d97e321c1a3a50116afbd189fd3fe03005338162c9d"
Nov 02 12:14:50.049387 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:14:50.049373    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dba20abb675323d063d4d97e321c1a3a50116afbd189fd3fe03005338162c9d\": container with ID starting with 6dba20abb675323d063d4d97e321c1a3a50116afbd189fd3fe03005338162c9d not found: ID does not exist" containerID="6dba20abb675323d063d4d97e321c1a3a50116afbd189fd3fe03005338162c9d"
Nov 02 12:14:50.049425 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.049392    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dba20abb675323d063d4d97e321c1a3a50116afbd189fd3fe03005338162c9d"} err="failed to get container status \"6dba20abb675323d063d4d97e321c1a3a50116afbd189fd3fe03005338162c9d\": rpc error: code = NotFound desc = could not find container \"6dba20abb675323d063d4d97e321c1a3a50116afbd189fd3fe03005338162c9d\": container with ID starting with 6dba20abb675323d063d4d97e321c1a3a50116afbd189fd3fe03005338162c9d not found: ID does not exist"
Nov 02 12:14:50.086331 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.086291    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-workload-certs\") pod \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") "
Nov 02 12:14:50.086514 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.086344    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-token\" (UniqueName: \"kubernetes.io/projected/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-token\") pod \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") "
Nov 02 12:14:50.086514 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.086375    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-podinfo\") pod \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") "
Nov 02 12:14:50.086514 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.086480    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-data\") pod \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") "
Nov 02 12:14:50.086680 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.086512    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-workload-socket\") pod \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") "
Nov 02 12:14:50.086680 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.086603    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istiod-ca-cert\") pod \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") "
Nov 02 12:14:50.086786 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.086695    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-credential-socket\") pod \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") "
Nov 02 12:14:50.086786 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.086766    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-envoy\") pod \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") "
Nov 02 12:14:50.086884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.086850    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-kube-api-access-zct88\" (UniqueName: \"kubernetes.io/projected/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-kube-api-access-zct88\") pod \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\" (UID: \"ad3f68ba-d0bc-4ae1-b9c1-88850db07172\") "
Nov 02 12:14:50.088300 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.088268    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-data" pod "ad3f68ba-d0bc-4ae1-b9c1-88850db07172" (UID: "ad3f68ba-d0bc-4ae1-b9c1-88850db07172"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:14:50.088440 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.088278    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-workload-socket" pod "ad3f68ba-d0bc-4ae1-b9c1-88850db07172" (UID: "ad3f68ba-d0bc-4ae1-b9c1-88850db07172"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:14:50.088503 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.088473    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-credential-socket" pod "ad3f68ba-d0bc-4ae1-b9c1-88850db07172" (UID: "ad3f68ba-d0bc-4ae1-b9c1-88850db07172"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:14:50.088503 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.088490    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-workload-certs" pod "ad3f68ba-d0bc-4ae1-b9c1-88850db07172" (UID: "ad3f68ba-d0bc-4ae1-b9c1-88850db07172"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:14:50.088616 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.088515    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istiod-ca-cert" pod "ad3f68ba-d0bc-4ae1-b9c1-88850db07172" (UID: "ad3f68ba-d0bc-4ae1-b9c1-88850db07172"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:14:50.090952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.090920    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-kube-api-access-zct88" pod "ad3f68ba-d0bc-4ae1-b9c1-88850db07172" (UID: "ad3f68ba-d0bc-4ae1-b9c1-88850db07172"). InnerVolumeSpecName "kube-api-access-zct88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:14:50.092005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.091455    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-token" pod "ad3f68ba-d0bc-4ae1-b9c1-88850db07172" (UID: "ad3f68ba-d0bc-4ae1-b9c1-88850db07172"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:14:50.092005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.091833    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-envoy" pod "ad3f68ba-d0bc-4ae1-b9c1-88850db07172" (UID: "ad3f68ba-d0bc-4ae1-b9c1-88850db07172"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:14:50.092165 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.092131    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-podinfo" pod "ad3f68ba-d0bc-4ae1-b9c1-88850db07172" (UID: "ad3f68ba-d0bc-4ae1-b9c1-88850db07172"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:14:50.188715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.188621    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:50.188715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.188647    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:50.188715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.188657    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zct88\" (UniqueName: \"kubernetes.io/projected/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-kube-api-access-zct88\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:50.188715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.188680    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:50.188715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.188692    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:50.188715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.188700    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:50.188715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.188709    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:50.188715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.188719    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:50.189288 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.188727    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad3f68ba-d0bc-4ae1-b9c1-88850db07172-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:50.456500 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.456384    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"]
Nov 02 12:14:50.478999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.469906    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-966817-openshift-default-5b7cfc678f-lf9f5"]
Nov 02 12:14:50.685481 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.685448    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"]
Nov 02 12:14:50.686178 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.686157    4464 cpu_manager.go:414] "containerMap: removing container" podUID="ad3f68ba-d0bc-4ae1-b9c1-88850db07172" containerName="istio-proxy"
Nov 02 12:14:50.686178 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.686178    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3f68ba-d0bc-4ae1-b9c1-88850db07172" containerName="istio-proxy"
Nov 02 12:14:50.686460 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.686206    4464 cpu_manager.go:414] "containerMap: removing container" podUID="ad3f68ba-d0bc-4ae1-b9c1-88850db07172" containerName="istio-proxy"
Nov 02 12:14:50.686460 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.686213    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3f68ba-d0bc-4ae1-b9c1-88850db07172" containerName="istio-proxy"
Nov 02 12:14:50.686460 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.686338    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3f68ba-d0bc-4ae1-b9c1-88850db07172" containerName="istio-proxy"
Nov 02 12:14:50.686460 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.686355    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3f68ba-d0bc-4ae1-b9c1-88850db07172" containerName="istio-proxy"
Nov 02 12:14:50.691611 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.691588    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"
Nov 02 12:14:50.735879 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.735856    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"]
Nov 02 12:14:50.746165 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.746140    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:14:50.746323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.746241    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:14:50.746323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.746276    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:14:50.746323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.746316    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:14:50.748229 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.748205    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"]
Nov 02 12:14:50.755997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.753212    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"
Nov 02 12:14:50.796311 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.796271    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22fc2f8-pf9f5\" (UID: \"f21b7ce9-f251-4c0e-b820-0af71bfd13ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"
Nov 02 12:14:50.796495 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.796474    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22fc2f8-pf9f5\" (UID: \"f21b7ce9-f251-4c0e-b820-0af71bfd13ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"
Nov 02 12:14:50.796551 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.796530    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj278\" (UniqueName: \"kubernetes.io/projected/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-kube-api-access-sj278\") pod \"maas-api-revoke-sub-f22fc2f8-pf9f5\" (UID: \"f21b7ce9-f251-4c0e-b820-0af71bfd13ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"
Nov 02 12:14:50.813175 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.813147    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"]
Nov 02 12:14:50.851378 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.851346    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 12:14:50.851516 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.851406    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9" podUID="cad06365-a977-4f66-94eb-2511cc1d02f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 12:14:50.898508 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.898477    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22fc2f8-pf9f5\" (UID: \"f21b7ce9-f251-4c0e-b820-0af71bfd13ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"
Nov 02 12:14:50.898695 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.898556    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj278\" (UniqueName: \"kubernetes.io/projected/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-kube-api-access-sj278\") pod \"maas-api-revoke-sub-f22fc2f8-pf9f5\" (UID: \"f21b7ce9-f251-4c0e-b820-0af71bfd13ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"
Nov 02 12:14:50.898695 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.898685    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz8xb\" (UniqueName: \"kubernetes.io/projected/a9377375-ca57-48ab-936b-b918a9fc457c-kube-api-access-pz8xb\") pod \"maas-api-revoke-sub-851771f0-jltsm\" (UID: \"a9377375-ca57-48ab-936b-b918a9fc457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"
Nov 02 12:14:50.898816 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.898717    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22fc2f8-pf9f5\" (UID: \"f21b7ce9-f251-4c0e-b820-0af71bfd13ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"
Nov 02 12:14:50.898894 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.898868    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9377375-ca57-48ab-936b-b918a9fc457c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-851771f0-jltsm\" (UID: \"a9377375-ca57-48ab-936b-b918a9fc457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"
Nov 02 12:14:50.898953 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.898929    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9377375-ca57-48ab-936b-b918a9fc457c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-851771f0-jltsm\" (UID: \"a9377375-ca57-48ab-936b-b918a9fc457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"
Nov 02 12:14:50.899126 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.899107    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f22fc2f8-pf9f5\" (UID: \"f21b7ce9-f251-4c0e-b820-0af71bfd13ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"
Nov 02 12:14:50.901106 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.901081    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f22fc2f8-pf9f5\" (UID: \"f21b7ce9-f251-4c0e-b820-0af71bfd13ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"
Nov 02 12:14:50.984961 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:50.984931    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj278\" (UniqueName: \"kubernetes.io/projected/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-kube-api-access-sj278\") pod \"maas-api-revoke-sub-f22fc2f8-pf9f5\" (UID: \"f21b7ce9-f251-4c0e-b820-0af71bfd13ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"
Nov 02 12:14:51.000658 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:51.000585    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz8xb\" (UniqueName: \"kubernetes.io/projected/a9377375-ca57-48ab-936b-b918a9fc457c-kube-api-access-pz8xb\") pod \"maas-api-revoke-sub-851771f0-jltsm\" (UID: \"a9377375-ca57-48ab-936b-b918a9fc457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"
Nov 02 12:14:51.000658 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:51.000649    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9377375-ca57-48ab-936b-b918a9fc457c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-851771f0-jltsm\" (UID: \"a9377375-ca57-48ab-936b-b918a9fc457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"
Nov 02 12:14:51.000859 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:51.000679    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9377375-ca57-48ab-936b-b918a9fc457c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-851771f0-jltsm\" (UID: \"a9377375-ca57-48ab-936b-b918a9fc457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"
Nov 02 12:14:51.001273 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:51.001252    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9377375-ca57-48ab-936b-b918a9fc457c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-851771f0-jltsm\" (UID: \"a9377375-ca57-48ab-936b-b918a9fc457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"
Nov 02 12:14:51.002913 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:51.002890    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9377375-ca57-48ab-936b-b918a9fc457c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-851771f0-jltsm\" (UID: \"a9377375-ca57-48ab-936b-b918a9fc457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"
Nov 02 12:14:51.005484 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:51.005463    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"
Nov 02 12:14:51.104802 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:51.104745    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz8xb\" (UniqueName: \"kubernetes.io/projected/a9377375-ca57-48ab-936b-b918a9fc457c-kube-api-access-pz8xb\") pod \"maas-api-revoke-sub-851771f0-jltsm\" (UID: \"a9377375-ca57-48ab-936b-b918a9fc457c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"
Nov 02 12:14:51.398714 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:51.398643    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"
Nov 02 12:14:51.413463 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:51.413438    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"]
Nov 02 12:14:51.415379 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:14:51.415340    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21b7ce9_f251_4c0e_b820_0af71bfd13ce.slice/crio-f099257532aa3899f5e5804d4edb50a55708195d846a78949610b968c09778f4 WatchSource:0}: Error finding container f099257532aa3899f5e5804d4edb50a55708195d846a78949610b968c09778f4: Status 404 returned error can't find the container with id f099257532aa3899f5e5804d4edb50a55708195d846a78949610b968c09778f4
Nov 02 12:14:51.582529 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:51.582497    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3f68ba-d0bc-4ae1-b9c1-88850db07172" path="/var/lib/kubelet/pods/ad3f68ba-d0bc-4ae1-b9c1-88850db07172/volumes"
Nov 02 12:14:51.783705 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:51.783679    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"]
Nov 02 12:14:51.784054 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:14:51.783929    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9377375_ca57_48ab_936b_b918a9fc457c.slice/crio-439e0f0d07f86e98b427eea009cbb5729b54471f8b1933853046a52d343dcab8 WatchSource:0}: Error finding container 439e0f0d07f86e98b427eea009cbb5729b54471f8b1933853046a52d343dcab8: Status 404 returned error can't find the container with id 439e0f0d07f86e98b427eea009cbb5729b54471f8b1933853046a52d343dcab8
Nov 02 12:14:51.852193 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:51.852167    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 12:14:51.852280 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:51.852216    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9" podUID="cad06365-a977-4f66-94eb-2511cc1d02f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 12:14:52.048941 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.048902    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm" event={"ID":"a9377375-ca57-48ab-936b-b918a9fc457c","Type":"ContainerStarted","Data":"439e0f0d07f86e98b427eea009cbb5729b54471f8b1933853046a52d343dcab8"}
Nov 02 12:14:52.050644 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.050610    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5" event={"ID":"f21b7ce9-f251-4c0e-b820-0af71bfd13ce","Type":"ContainerStarted","Data":"e6253d0e6275a71317cf57703e33262852f2f4618d67bc78ce0b64d52046a285"}
Nov 02 12:14:52.050789 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.050669    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5" event={"ID":"f21b7ce9-f251-4c0e-b820-0af71bfd13ce","Type":"ContainerStarted","Data":"f099257532aa3899f5e5804d4edb50a55708195d846a78949610b968c09778f4"}
Nov 02 12:14:52.171890 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.171846    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5" podStartSLOduration=2.171833243 podStartE2EDuration="2.171833243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:14:52.170888044 +0000 UTC m=+1120.734596454" watchObservedRunningTime="2026-11-02 12:14:52.171833243 +0000 UTC m=+1120.735541651"
Nov 02 12:14:52.442151 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.442130    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:14:52.522687 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.522623    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-envoy\") pod \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") "
Nov 02 12:14:52.522807 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.522700    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-kube-api-access-lvfc9\" (UniqueName: \"kubernetes.io/projected/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-kube-api-access-lvfc9\") pod \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") "
Nov 02 12:14:52.522807 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.522734    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-data\") pod \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") "
Nov 02 12:14:52.522885 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.522806    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-credential-socket\") pod \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") "
Nov 02 12:14:52.522885 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.522854    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-token\" (UniqueName: \"kubernetes.io/projected/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-token\") pod \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") "
Nov 02 12:14:52.523006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.522893    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-workload-certs\") pod \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") "
Nov 02 12:14:52.523006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.522924    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-podinfo\") pod \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") "
Nov 02 12:14:52.523006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.522963    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-workload-socket\") pod \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") "
Nov 02 12:14:52.523164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.523035    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istiod-ca-cert\") pod \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\" (UID: \"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4\") "
Nov 02 12:14:52.523164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.523128    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-credential-socket" pod "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" (UID: "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:14:52.523361 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.523322    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-workload-certs" pod "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" (UID: "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:14:52.523449 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.523432    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-data" pod "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" (UID: "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:14:52.523514 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.523466    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istiod-ca-cert" pod "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" (UID: "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:14:52.523514 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.523478    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-workload-socket" pod "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" (UID: "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:14:52.523623 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.523579    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:52.523623 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.523599    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:52.523623 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.523615    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:52.523777 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.523630    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:52.523777 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.523645    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:52.524953 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.524927    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-kube-api-access-lvfc9" pod "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" (UID: "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4"). InnerVolumeSpecName "kube-api-access-lvfc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:14:52.525100 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.525004    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-envoy" pod "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" (UID: "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:14:52.525265 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.525244    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-token" pod "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" (UID: "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:14:52.525314 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.525282    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-podinfo" pod "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" (UID: "4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:14:52.625529 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.625327    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:52.625529 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.625360    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:52.625529 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.625377    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:52.625529 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.625392    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvfc9\" (UniqueName: \"kubernetes.io/projected/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4-kube-api-access-lvfc9\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:52.854857 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.854779    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:52.855334 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.855295    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:52.856048 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:52.856026    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:14:53.058682 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:53.058651    4464 generic.go:356] "Generic (PLEG): container finished" podID="4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" containerID="6bf1c07240323add86bb44f0127a2f664f57b5a2afdc5eacbe75cbf2154e5aa3" exitCode=0
Nov 02 12:14:53.058865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:53.058701    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"
Nov 02 12:14:53.058865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:53.058741    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn" event={"ID":"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4","Type":"ContainerDied","Data":"6bf1c07240323add86bb44f0127a2f664f57b5a2afdc5eacbe75cbf2154e5aa3"}
Nov 02 12:14:53.058865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:53.058801    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn" event={"ID":"4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4","Type":"ContainerDied","Data":"bc04544cacb0a643044c4d7926c9b3abff7dfaa0f7119c9d148716d0cad388e8"}
Nov 02 12:14:53.058865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:53.058834    4464 scope.go:122] "RemoveContainer" containerID="6bf1c07240323add86bb44f0127a2f664f57b5a2afdc5eacbe75cbf2154e5aa3"
Nov 02 12:14:53.061724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:53.061699    4464 generic.go:356] "Generic (PLEG): container finished" podID="a9377375-ca57-48ab-936b-b918a9fc457c" containerID="983ba93327f9001dc788e6901c7221ecbceee546733770cad30c6ebf122e1c78" exitCode=0
Nov 02 12:14:53.061849 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:53.061798    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm" event={"ID":"a9377375-ca57-48ab-936b-b918a9fc457c","Type":"ContainerDied","Data":"983ba93327f9001dc788e6901c7221ecbceee546733770cad30c6ebf122e1c78"}
Nov 02 12:14:53.063766 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:53.063742    4464 generic.go:356] "Generic (PLEG): container finished" podID="f21b7ce9-f251-4c0e-b820-0af71bfd13ce" containerID="e6253d0e6275a71317cf57703e33262852f2f4618d67bc78ce0b64d52046a285" exitCode=0
Nov 02 12:14:53.063873 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:53.063826    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5" event={"ID":"f21b7ce9-f251-4c0e-b820-0af71bfd13ce","Type":"ContainerDied","Data":"e6253d0e6275a71317cf57703e33262852f2f4618d67bc78ce0b64d52046a285"}
Nov 02 12:14:53.071813 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:53.071617    4464 scope.go:122] "RemoveContainer" containerID="6bf1c07240323add86bb44f0127a2f664f57b5a2afdc5eacbe75cbf2154e5aa3"
Nov 02 12:14:53.072014 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:14:53.071962    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf1c07240323add86bb44f0127a2f664f57b5a2afdc5eacbe75cbf2154e5aa3\": container with ID starting with 6bf1c07240323add86bb44f0127a2f664f57b5a2afdc5eacbe75cbf2154e5aa3 not found: ID does not exist" containerID="6bf1c07240323add86bb44f0127a2f664f57b5a2afdc5eacbe75cbf2154e5aa3"
Nov 02 12:14:53.072132 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:53.072027    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf1c07240323add86bb44f0127a2f664f57b5a2afdc5eacbe75cbf2154e5aa3"} err="failed to get container status \"6bf1c07240323add86bb44f0127a2f664f57b5a2afdc5eacbe75cbf2154e5aa3\": rpc error: code = NotFound desc = could not find container \"6bf1c07240323add86bb44f0127a2f664f57b5a2afdc5eacbe75cbf2154e5aa3\": container with ID starting with 6bf1c07240323add86bb44f0127a2f664f57b5a2afdc5eacbe75cbf2154e5aa3 not found: ID does not exist"
Nov 02 12:14:53.363645 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:53.363615    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"]
Nov 02 12:14:53.368020 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:53.367995    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ddhbn"]
Nov 02 12:14:53.582335 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:53.582301    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" path="/var/lib/kubelet/pods/4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4/volumes"
Nov 02 12:14:54.223652 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.223627    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"
Nov 02 12:14:54.245493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.245471    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"
Nov 02 12:14:54.246838 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.246820    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-maas-api-xks-ca\") pod \"f21b7ce9-f251-4c0e-b820-0af71bfd13ce\" (UID: \"f21b7ce9-f251-4c0e-b820-0af71bfd13ce\") "
Nov 02 12:14:54.247006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.246967    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-kube-api-access-sj278\" (UniqueName: \"kubernetes.io/projected/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-kube-api-access-sj278\") pod \"f21b7ce9-f251-4c0e-b820-0af71bfd13ce\" (UID: \"f21b7ce9-f251-4c0e-b820-0af71bfd13ce\") "
Nov 02 12:14:54.247065 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.247058    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-maas-api-openshift-ca\") pod \"f21b7ce9-f251-4c0e-b820-0af71bfd13ce\" (UID: \"f21b7ce9-f251-4c0e-b820-0af71bfd13ce\") "
Nov 02 12:14:54.247371 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.247349    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-maas-api-openshift-ca" pod "f21b7ce9-f251-4c0e-b820-0af71bfd13ce" (UID: "f21b7ce9-f251-4c0e-b820-0af71bfd13ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:14:54.247474 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.247454    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:54.248855 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.248838    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-maas-api-xks-ca" pod "f21b7ce9-f251-4c0e-b820-0af71bfd13ce" (UID: "f21b7ce9-f251-4c0e-b820-0af71bfd13ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:14:54.248933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.248861    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-kube-api-access-sj278" pod "f21b7ce9-f251-4c0e-b820-0af71bfd13ce" (UID: "f21b7ce9-f251-4c0e-b820-0af71bfd13ce"). InnerVolumeSpecName "kube-api-access-sj278". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:14:54.348813 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.348755    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9377375-ca57-48ab-936b-b918a9fc457c-kube-api-access-pz8xb\" (UniqueName: \"kubernetes.io/projected/a9377375-ca57-48ab-936b-b918a9fc457c-kube-api-access-pz8xb\") pod \"a9377375-ca57-48ab-936b-b918a9fc457c\" (UID: \"a9377375-ca57-48ab-936b-b918a9fc457c\") "
Nov 02 12:14:54.348918 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.348833    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9377375-ca57-48ab-936b-b918a9fc457c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9377375-ca57-48ab-936b-b918a9fc457c-maas-api-openshift-ca\") pod \"a9377375-ca57-48ab-936b-b918a9fc457c\" (UID: \"a9377375-ca57-48ab-936b-b918a9fc457c\") "
Nov 02 12:14:54.349003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.348949    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9377375-ca57-48ab-936b-b918a9fc457c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9377375-ca57-48ab-936b-b918a9fc457c-maas-api-xks-ca\") pod \"a9377375-ca57-48ab-936b-b918a9fc457c\" (UID: \"a9377375-ca57-48ab-936b-b918a9fc457c\") "
Nov 02 12:14:54.349223 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.349202    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9377375-ca57-48ab-936b-b918a9fc457c-maas-api-openshift-ca" pod "a9377375-ca57-48ab-936b-b918a9fc457c" (UID: "a9377375-ca57-48ab-936b-b918a9fc457c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:14:54.349282 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.349229    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:54.349282 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.349242    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj278\" (UniqueName: \"kubernetes.io/projected/f21b7ce9-f251-4c0e-b820-0af71bfd13ce-kube-api-access-sj278\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:54.350776 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.350756    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9377375-ca57-48ab-936b-b918a9fc457c-maas-api-xks-ca" pod "a9377375-ca57-48ab-936b-b918a9fc457c" (UID: "a9377375-ca57-48ab-936b-b918a9fc457c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:14:54.350826 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.350806    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9377375-ca57-48ab-936b-b918a9fc457c-kube-api-access-pz8xb" pod "a9377375-ca57-48ab-936b-b918a9fc457c" (UID: "a9377375-ca57-48ab-936b-b918a9fc457c"). InnerVolumeSpecName "kube-api-access-pz8xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:14:54.450660 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.450630    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9377375-ca57-48ab-936b-b918a9fc457c-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:54.450660 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.450656    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz8xb\" (UniqueName: \"kubernetes.io/projected/a9377375-ca57-48ab-936b-b918a9fc457c-kube-api-access-pz8xb\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:54.450849 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:54.450667    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9377375-ca57-48ab-936b-b918a9fc457c-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:14:55.080414 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:55.080382    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"
Nov 02 12:14:55.080607 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:55.080422    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5" event={"ID":"f21b7ce9-f251-4c0e-b820-0af71bfd13ce","Type":"ContainerDied","Data":"f099257532aa3899f5e5804d4edb50a55708195d846a78949610b968c09778f4"}
Nov 02 12:14:55.080607 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:55.080476    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f099257532aa3899f5e5804d4edb50a55708195d846a78949610b968c09778f4"
Nov 02 12:14:55.082047 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:55.082023    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm" event={"ID":"a9377375-ca57-48ab-936b-b918a9fc457c","Type":"ContainerDied","Data":"439e0f0d07f86e98b427eea009cbb5729b54471f8b1933853046a52d343dcab8"}
Nov 02 12:14:55.082177 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:55.082063    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="439e0f0d07f86e98b427eea009cbb5729b54471f8b1933853046a52d343dcab8"
Nov 02 12:14:55.082177 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:55.082071    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"
Nov 02 12:14:55.868033 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:14:55.861654    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:14:58.502000 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:14:58.498093    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:15:00.192394 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.192360    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989"]
Nov 02 12:15:00.193195 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.193165    4464 cpu_manager.go:414] "containerMap: removing container" podUID="4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" containerName="istio-proxy"
Nov 02 12:15:00.193195 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.193187    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" containerName="istio-proxy"
Nov 02 12:15:00.193417 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.193214    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f21b7ce9-f251-4c0e-b820-0af71bfd13ce" containerName="revoke-keys"
Nov 02 12:15:00.193417 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.193220    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f21b7ce9-f251-4c0e-b820-0af71bfd13ce" containerName="revoke-keys"
Nov 02 12:15:00.193417 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.193233    4464 cpu_manager.go:414] "containerMap: removing container" podUID="a9377375-ca57-48ab-936b-b918a9fc457c" containerName="revoke-keys"
Nov 02 12:15:00.193417 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.193239    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9377375-ca57-48ab-936b-b918a9fc457c" containerName="revoke-keys"
Nov 02 12:15:00.193417 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.193330    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f21b7ce9-f251-4c0e-b820-0af71bfd13ce" containerName="revoke-keys"
Nov 02 12:15:00.193417 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.193339    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9377375-ca57-48ab-936b-b918a9fc457c" containerName="revoke-keys"
Nov 02 12:15:00.193417 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.193351    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="4888bcb6-bcf0-4c46-9e89-bc4f1feb6df4" containerName="istio-proxy"
Nov 02 12:15:00.198478 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.198453    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv"]
Nov 02 12:15:00.198605 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.198579    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989"
Nov 02 12:15:00.203192 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.203175    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf"]
Nov 02 12:15:00.203297 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.203282    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv"
Nov 02 12:15:00.206902 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.206884    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf"
Nov 02 12:15:00.236217 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.236194    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8"]
Nov 02 12:15:00.239950 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.239932    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt"]
Nov 02 12:15:00.240036 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.240000    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8"
Nov 02 12:15:00.244264 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.244249    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt"
Nov 02 12:15:00.253319 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.253298    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:15:00.253410 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.253395    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:15:00.297193 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.297160    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv"]
Nov 02 12:15:00.303942 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.303911    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989"]
Nov 02 12:15:00.343952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.343910    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:15:00.344231 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.344211    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf"]
Nov 02 12:15:00.349100 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.349061    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f99e5052-6768-4b52-ba63-dd93eeae9a99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv\" (UID: \"f99e5052-6768-4b52-ba63-dd93eeae9a99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv"
Nov 02 12:15:00.349219 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.349145    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/176450dd-cb60-4c5c-9f12-110ed3091333-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989\" (UID: \"176450dd-cb60-4c5c-9f12-110ed3091333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989"
Nov 02 12:15:00.349219 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.349190    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cae5d40-15f7-4c11-91a2-74d88328fecf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt\" (UID: \"6cae5d40-15f7-4c11-91a2-74d88328fecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt"
Nov 02 12:15:00.349293 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.349235    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkhrm\" (UniqueName: \"kubernetes.io/projected/6cae5d40-15f7-4c11-91a2-74d88328fecf-kube-api-access-kkhrm\") pod \"maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt\" (UID: \"6cae5d40-15f7-4c11-91a2-74d88328fecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt"
Nov 02 12:15:00.349338 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.349322    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdpkt\" (UniqueName: \"kubernetes.io/projected/176450dd-cb60-4c5c-9f12-110ed3091333-kube-api-access-hdpkt\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989\" (UID: \"176450dd-cb60-4c5c-9f12-110ed3091333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989"
Nov 02 12:15:00.349375 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.349356    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23be3d03-23fc-418b-8d80-58736eb9217e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893695-5hxmf\" (UID: \"23be3d03-23fc-418b-8d80-58736eb9217e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf"
Nov 02 12:15:00.349412 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.349383    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8\" (UID: \"f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8"
Nov 02 12:15:00.349412 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.349406    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbfjn\" (UniqueName: \"kubernetes.io/projected/f99e5052-6768-4b52-ba63-dd93eeae9a99-kube-api-access-wbfjn\") pod \"maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv\" (UID: \"f99e5052-6768-4b52-ba63-dd93eeae9a99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv"
Nov 02 12:15:00.349472 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.349445    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkhft\" (UniqueName: \"kubernetes.io/projected/23be3d03-23fc-418b-8d80-58736eb9217e-kube-api-access-xkhft\") pod \"maas-api-key-cleanup-29893695-5hxmf\" (UID: \"23be3d03-23fc-418b-8d80-58736eb9217e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf"
Nov 02 12:15:00.349516 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.349469    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7j67\" (UniqueName: \"kubernetes.io/projected/f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6-kube-api-access-n7j67\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8\" (UID: \"f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8"
Nov 02 12:15:00.349684 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.349667    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8"]
Nov 02 12:15:00.353628 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.353602    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt"]
Nov 02 12:15:00.451079 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.450968    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdpkt\" (UniqueName: \"kubernetes.io/projected/176450dd-cb60-4c5c-9f12-110ed3091333-kube-api-access-hdpkt\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989\" (UID: \"176450dd-cb60-4c5c-9f12-110ed3091333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989"
Nov 02 12:15:00.451079 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.451032    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23be3d03-23fc-418b-8d80-58736eb9217e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893695-5hxmf\" (UID: \"23be3d03-23fc-418b-8d80-58736eb9217e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf"
Nov 02 12:15:00.451079 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.451054    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8\" (UID: \"f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8"
Nov 02 12:15:00.451079 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.451075    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbfjn\" (UniqueName: \"kubernetes.io/projected/f99e5052-6768-4b52-ba63-dd93eeae9a99-kube-api-access-wbfjn\") pod \"maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv\" (UID: \"f99e5052-6768-4b52-ba63-dd93eeae9a99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv"
Nov 02 12:15:00.451395 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.451104    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkhft\" (UniqueName: \"kubernetes.io/projected/23be3d03-23fc-418b-8d80-58736eb9217e-kube-api-access-xkhft\") pod \"maas-api-key-cleanup-29893695-5hxmf\" (UID: \"23be3d03-23fc-418b-8d80-58736eb9217e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf"
Nov 02 12:15:00.451395 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.451128    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7j67\" (UniqueName: \"kubernetes.io/projected/f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6-kube-api-access-n7j67\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8\" (UID: \"f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8"
Nov 02 12:15:00.451395 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.451247    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f99e5052-6768-4b52-ba63-dd93eeae9a99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv\" (UID: \"f99e5052-6768-4b52-ba63-dd93eeae9a99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv"
Nov 02 12:15:00.451395 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.451327    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/176450dd-cb60-4c5c-9f12-110ed3091333-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989\" (UID: \"176450dd-cb60-4c5c-9f12-110ed3091333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989"
Nov 02 12:15:00.451395 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.451378    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cae5d40-15f7-4c11-91a2-74d88328fecf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt\" (UID: \"6cae5d40-15f7-4c11-91a2-74d88328fecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt"
Nov 02 12:15:00.451675 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.451431    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhrm\" (UniqueName: \"kubernetes.io/projected/6cae5d40-15f7-4c11-91a2-74d88328fecf-kube-api-access-kkhrm\") pod \"maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt\" (UID: \"6cae5d40-15f7-4c11-91a2-74d88328fecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt"
Nov 02 12:15:00.451821 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.451800    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23be3d03-23fc-418b-8d80-58736eb9217e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893695-5hxmf\" (UID: \"23be3d03-23fc-418b-8d80-58736eb9217e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf"
Nov 02 12:15:00.451888 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.451871    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8\" (UID: \"f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8"
Nov 02 12:15:00.452030 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.452013    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/176450dd-cb60-4c5c-9f12-110ed3091333-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989\" (UID: \"176450dd-cb60-4c5c-9f12-110ed3091333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989"
Nov 02 12:15:00.452078 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.452041    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f99e5052-6768-4b52-ba63-dd93eeae9a99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv\" (UID: \"f99e5052-6768-4b52-ba63-dd93eeae9a99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv"
Nov 02 12:15:00.452110 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.452092    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cae5d40-15f7-4c11-91a2-74d88328fecf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt\" (UID: \"6cae5d40-15f7-4c11-91a2-74d88328fecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt"
Nov 02 12:15:00.512551 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.512528    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkhft\" (UniqueName: \"kubernetes.io/projected/23be3d03-23fc-418b-8d80-58736eb9217e-kube-api-access-xkhft\") pod \"maas-api-key-cleanup-29893695-5hxmf\" (UID: \"23be3d03-23fc-418b-8d80-58736eb9217e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf"
Nov 02 12:15:00.512678 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.512652    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdpkt\" (UniqueName: \"kubernetes.io/projected/176450dd-cb60-4c5c-9f12-110ed3091333-kube-api-access-hdpkt\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989\" (UID: \"176450dd-cb60-4c5c-9f12-110ed3091333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989"
Nov 02 12:15:00.512825 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.512801    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbfjn\" (UniqueName: \"kubernetes.io/projected/f99e5052-6768-4b52-ba63-dd93eeae9a99-kube-api-access-wbfjn\") pod \"maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv\" (UID: \"f99e5052-6768-4b52-ba63-dd93eeae9a99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv"
Nov 02 12:15:00.513039 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.513020    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7j67\" (UniqueName: \"kubernetes.io/projected/f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6-kube-api-access-n7j67\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8\" (UID: \"f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8"
Nov 02 12:15:00.513106 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.513067    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkhrm\" (UniqueName: \"kubernetes.io/projected/6cae5d40-15f7-4c11-91a2-74d88328fecf-kube-api-access-kkhrm\") pod \"maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt\" (UID: \"6cae5d40-15f7-4c11-91a2-74d88328fecf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt"
Nov 02 12:15:00.513754 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.513738    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989"
Nov 02 12:15:00.521881 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.521866    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv"
Nov 02 12:15:00.528105 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.528086    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf"
Nov 02 12:15:00.555870 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.555260    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8"
Nov 02 12:15:00.561898 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.561872    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt"
Nov 02 12:15:00.900011 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:15:00.899968    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176450dd_cb60_4c5c_9f12_110ed3091333.slice/crio-593152a548d3a0b142424f4e60d4ea2b0315c71fa21bd7c095552b4397e387d3 WatchSource:0}: Error finding container 593152a548d3a0b142424f4e60d4ea2b0315c71fa21bd7c095552b4397e387d3: Status 404 returned error can't find the container with id 593152a548d3a0b142424f4e60d4ea2b0315c71fa21bd7c095552b4397e387d3
Nov 02 12:15:00.900263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.900239    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989"]
Nov 02 12:15:00.951949 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.951922    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv"]
Nov 02 12:15:00.952290 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:15:00.952265    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99e5052_6768_4b52_ba63_dd93eeae9a99.slice/crio-ca3ffc0416ca7a69a1272943858512e6df723f5c07aa002770373203d6f9f368 WatchSource:0}: Error finding container ca3ffc0416ca7a69a1272943858512e6df723f5c07aa002770373203d6f9f368: Status 404 returned error can't find the container with id ca3ffc0416ca7a69a1272943858512e6df723f5c07aa002770373203d6f9f368
Nov 02 12:15:00.952734 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:15:00.952720    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23be3d03_23fc_418b_8d80_58736eb9217e.slice/crio-a0790367ff8c05e87655b1aae2b29a179f8552834d5573349361ed6b7f710aa3 WatchSource:0}: Error finding container a0790367ff8c05e87655b1aae2b29a179f8552834d5573349361ed6b7f710aa3: Status 404 returned error can't find the container with id a0790367ff8c05e87655b1aae2b29a179f8552834d5573349361ed6b7f710aa3
Nov 02 12:15:00.956296 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:00.956274    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf"]
Nov 02 12:15:01.008388 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:15:01.008365    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cae5d40_15f7_4c11_91a2_74d88328fecf.slice/crio-11661a60a32403bb1c7d2deee60ceb8cacbf5ca8ede90d403cf175d93ac24528 WatchSource:0}: Error finding container 11661a60a32403bb1c7d2deee60ceb8cacbf5ca8ede90d403cf175d93ac24528: Status 404 returned error can't find the container with id 11661a60a32403bb1c7d2deee60ceb8cacbf5ca8ede90d403cf175d93ac24528
Nov 02 12:15:01.009335 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:01.009063    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt"]
Nov 02 12:15:01.013349 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:15:01.013326    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43c5b36_e9bb_4646_bec4_e2ccf98b7bb6.slice/crio-fdd347ffdf907e17f309c0b7afa11d9f974da45455b0caf3e95b00413a46334d WatchSource:0}: Error finding container fdd347ffdf907e17f309c0b7afa11d9f974da45455b0caf3e95b00413a46334d: Status 404 returned error can't find the container with id fdd347ffdf907e17f309c0b7afa11d9f974da45455b0caf3e95b00413a46334d
Nov 02 12:15:01.013914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:01.013890    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8"]
Nov 02 12:15:01.452704 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:01.452625    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8" event={"ID":"f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6","Type":"ContainerStarted","Data":"fdd347ffdf907e17f309c0b7afa11d9f974da45455b0caf3e95b00413a46334d"}
Nov 02 12:15:01.454186 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:01.454161    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv" event={"ID":"f99e5052-6768-4b52-ba63-dd93eeae9a99","Type":"ContainerStarted","Data":"0e2694d7b273ed195b46bf89b5ac556508a7942ca1fd249abca4f594bff13aa3"}
Nov 02 12:15:01.454327 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:01.454206    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv" event={"ID":"f99e5052-6768-4b52-ba63-dd93eeae9a99","Type":"ContainerStarted","Data":"ca3ffc0416ca7a69a1272943858512e6df723f5c07aa002770373203d6f9f368"}
Nov 02 12:15:01.455741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:01.455714    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989" event={"ID":"176450dd-cb60-4c5c-9f12-110ed3091333","Type":"ContainerStarted","Data":"59d2d52c51099590a4997f995d1d3b7a780299af8c9d52725f734590b2f98d2f"}
Nov 02 12:15:01.455848 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:01.455761    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989" event={"ID":"176450dd-cb60-4c5c-9f12-110ed3091333","Type":"ContainerStarted","Data":"593152a548d3a0b142424f4e60d4ea2b0315c71fa21bd7c095552b4397e387d3"}
Nov 02 12:15:01.456924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:01.456900    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt" event={"ID":"6cae5d40-15f7-4c11-91a2-74d88328fecf","Type":"ContainerStarted","Data":"11661a60a32403bb1c7d2deee60ceb8cacbf5ca8ede90d403cf175d93ac24528"}
Nov 02 12:15:01.458015 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:01.457995    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf" event={"ID":"23be3d03-23fc-418b-8d80-58736eb9217e","Type":"ContainerStarted","Data":"a0790367ff8c05e87655b1aae2b29a179f8552834d5573349361ed6b7f710aa3"}
Nov 02 12:15:01.575365 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:01.575315    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv" podStartSLOduration=1.575299293 podStartE2EDuration="1.575299293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:15:01.573171523 +0000 UTC m=+1130.136879933" watchObservedRunningTime="2026-11-02 12:15:01.575299293 +0000 UTC m=+1130.139007702"
Nov 02 12:15:01.691215 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:01.691165    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989" podStartSLOduration=1.6911525059999999 podStartE2EDuration="1.691152506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:15:01.689613409 +0000 UTC m=+1130.253321818" watchObservedRunningTime="2026-11-02 12:15:01.691152506 +0000 UTC m=+1130.254860940"
Nov 02 12:15:02.082684 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.082660    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"]
Nov 02 12:15:02.086826 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.086808    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.140300 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.140278    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-dd087a-openshift-default-dockercfg-7896p\""
Nov 02 12:15:02.149155 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.149130    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"]
Nov 02 12:15:02.175236 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.175203    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-credential-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.175384 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.175254    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tblrj\" (UniqueName: \"kubernetes.io/projected/4b684afa-613a-43b5-b10d-0f0ece8da04c-kube-api-access-tblrj\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.175384 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.175282    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-workload-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.175517 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.175420    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b684afa-613a-43b5-b10d-0f0ece8da04c-istiod-ca-cert\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.175572 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.175525    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-workload-certs\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.175572 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.175567    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-podinfo\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.175676 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.175590    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-token\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.175728 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.175674    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-data\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.175728 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.175721    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-envoy\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.277066 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.277035    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-podinfo\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.277163 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.277078    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-token\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.277163 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.277126    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-data\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.277163 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.277152    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-envoy\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.277292 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.277184    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-credential-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.277292 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.277221    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tblrj\" (UniqueName: \"kubernetes.io/projected/4b684afa-613a-43b5-b10d-0f0ece8da04c-kube-api-access-tblrj\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.277292 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.277250    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-workload-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.277418 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.277305    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b684afa-613a-43b5-b10d-0f0ece8da04c-istiod-ca-cert\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.277576 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.277552    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-workload-certs\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.277749 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.277728    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-data\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.277819 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.277791    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-credential-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.277874 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.277851    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-workload-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.278061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.278041    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-workload-certs\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.278221 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.278207    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b684afa-613a-43b5-b10d-0f0ece8da04c-istiod-ca-cert\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.279435 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.279416    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-envoy\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.279695 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.279678    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-podinfo\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.339438 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.339377    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-token\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.339521 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.339479    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tblrj\" (UniqueName: \"kubernetes.io/projected/4b684afa-613a-43b5-b10d-0f0ece8da04c-kube-api-access-tblrj\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:02.467302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.467271    4464 generic.go:356] "Generic (PLEG): container finished" podID="6cae5d40-15f7-4c11-91a2-74d88328fecf" containerID="1ed16ec831932e7de2bdb097e78f3e88f680625ece4ac77d64abd9793e9ba1ae" exitCode=60
Nov 02 12:15:02.467709 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.467323    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt" event={"ID":"6cae5d40-15f7-4c11-91a2-74d88328fecf","Type":"ContainerDied","Data":"1ed16ec831932e7de2bdb097e78f3e88f680625ece4ac77d64abd9793e9ba1ae"}
Nov 02 12:15:02.467709 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.467618    4464 scope.go:122] "RemoveContainer" containerID="1ed16ec831932e7de2bdb097e78f3e88f680625ece4ac77d64abd9793e9ba1ae"
Nov 02 12:15:02.469149 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.469111    4464 generic.go:356] "Generic (PLEG): container finished" podID="23be3d03-23fc-418b-8d80-58736eb9217e" containerID="d2c6725877202d85dd8585d64897da90b441e1e868ee1156f1854bc7028d95cc" exitCode=60
Nov 02 12:15:02.469252 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.469225    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf" event={"ID":"23be3d03-23fc-418b-8d80-58736eb9217e","Type":"ContainerDied","Data":"d2c6725877202d85dd8585d64897da90b441e1e868ee1156f1854bc7028d95cc"}
Nov 02 12:15:02.469404 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.469387    4464 scope.go:122] "RemoveContainer" containerID="d2c6725877202d85dd8585d64897da90b441e1e868ee1156f1854bc7028d95cc"
Nov 02 12:15:02.471221 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.471200    4464 generic.go:356] "Generic (PLEG): container finished" podID="f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6" containerID="aa7dbac2638f17a5d12cae2e0b25c4de6d39c776b3250de795b2a5682a5ab70c" exitCode=60
Nov 02 12:15:02.471330 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.471292    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8" event={"ID":"f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6","Type":"ContainerDied","Data":"aa7dbac2638f17a5d12cae2e0b25c4de6d39c776b3250de795b2a5682a5ab70c"}
Nov 02 12:15:02.471461 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.471444    4464 scope.go:122] "RemoveContainer" containerID="aa7dbac2638f17a5d12cae2e0b25c4de6d39c776b3250de795b2a5682a5ab70c"
Nov 02 12:15:02.472890 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.472867    4464 generic.go:356] "Generic (PLEG): container finished" podID="f99e5052-6768-4b52-ba63-dd93eeae9a99" containerID="0e2694d7b273ed195b46bf89b5ac556508a7942ca1fd249abca4f594bff13aa3" exitCode=60
Nov 02 12:15:02.472999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.472946    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv" event={"ID":"f99e5052-6768-4b52-ba63-dd93eeae9a99","Type":"ContainerDied","Data":"0e2694d7b273ed195b46bf89b5ac556508a7942ca1fd249abca4f594bff13aa3"}
Nov 02 12:15:02.473151 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.473127    4464 scope.go:122] "RemoveContainer" containerID="0e2694d7b273ed195b46bf89b5ac556508a7942ca1fd249abca4f594bff13aa3"
Nov 02 12:15:02.474610 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.474582    4464 generic.go:356] "Generic (PLEG): container finished" podID="176450dd-cb60-4c5c-9f12-110ed3091333" containerID="59d2d52c51099590a4997f995d1d3b7a780299af8c9d52725f734590b2f98d2f" exitCode=60
Nov 02 12:15:02.474704 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.474632    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989" event={"ID":"176450dd-cb60-4c5c-9f12-110ed3091333","Type":"ContainerDied","Data":"59d2d52c51099590a4997f995d1d3b7a780299af8c9d52725f734590b2f98d2f"}
Nov 02 12:15:02.474847 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.474832    4464 scope.go:122] "RemoveContainer" containerID="59d2d52c51099590a4997f995d1d3b7a780299af8c9d52725f734590b2f98d2f"
Nov 02 12:15:02.612173 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:02.612105    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:03.083041 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:15:03.083008    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b684afa_613a_43b5_b10d_0f0ece8da04c.slice/crio-f17432f849c61a5a1276a1f7df1745c3425d325db4f45207319a2d70a334ea5d WatchSource:0}: Error finding container f17432f849c61a5a1276a1f7df1745c3425d325db4f45207319a2d70a334ea5d: Status 404 returned error can't find the container with id f17432f849c61a5a1276a1f7df1745c3425d325db4f45207319a2d70a334ea5d
Nov 02 12:15:03.083326 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.083304    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"]
Nov 02 12:15:03.085501 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.085471    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:15:03.085629 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.085558    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:15:03.085629 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.085601    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:15:03.483840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.483810    4464 generic.go:356] "Generic (PLEG): container finished" podID="f99e5052-6768-4b52-ba63-dd93eeae9a99" containerID="ded45c1e38c042aea8d33bfd73d8905f222d4855d6526c5ead281359a0bc9dd8" exitCode=60
Nov 02 12:15:03.484275 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.483887    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv" event={"ID":"f99e5052-6768-4b52-ba63-dd93eeae9a99","Type":"ContainerDied","Data":"ded45c1e38c042aea8d33bfd73d8905f222d4855d6526c5ead281359a0bc9dd8"}
Nov 02 12:15:03.484275 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.483959    4464 scope.go:122] "RemoveContainer" containerID="0e2694d7b273ed195b46bf89b5ac556508a7942ca1fd249abca4f594bff13aa3"
Nov 02 12:15:03.484389 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.484287    4464 scope.go:122] "RemoveContainer" containerID="ded45c1e38c042aea8d33bfd73d8905f222d4855d6526c5ead281359a0bc9dd8"
Nov 02 12:15:03.484663 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:15:03.484635    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv_odh-ai-gateway-infra(f99e5052-6768-4b52-ba63-dd93eeae9a99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv" podUID="f99e5052-6768-4b52-ba63-dd93eeae9a99"
Nov 02 12:15:03.485848 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.485821    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb" event={"ID":"4b684afa-613a-43b5-b10d-0f0ece8da04c","Type":"ContainerStarted","Data":"da2129fb600ab49a1312806990197fed9e3a62e365bf957e8ca2d2d920e85618"}
Nov 02 12:15:03.485962 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.485875    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb" event={"ID":"4b684afa-613a-43b5-b10d-0f0ece8da04c","Type":"ContainerStarted","Data":"f17432f849c61a5a1276a1f7df1745c3425d325db4f45207319a2d70a334ea5d"}
Nov 02 12:15:03.487928 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.487902    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989" event={"ID":"176450dd-cb60-4c5c-9f12-110ed3091333","Type":"ContainerStarted","Data":"95b26fb4cbdf5564ef97a977640238ede3849af6c8df6476ba61f9443b2c8233"}
Nov 02 12:15:03.489893 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.489869    4464 generic.go:356] "Generic (PLEG): container finished" podID="6cae5d40-15f7-4c11-91a2-74d88328fecf" containerID="726e8aee3bae429127c2da816464b92aa8ce4e63ab8b60ff798eb89e444e0f18" exitCode=60
Nov 02 12:15:03.490005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.489947    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt" event={"ID":"6cae5d40-15f7-4c11-91a2-74d88328fecf","Type":"ContainerDied","Data":"726e8aee3bae429127c2da816464b92aa8ce4e63ab8b60ff798eb89e444e0f18"}
Nov 02 12:15:03.490215 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.490197    4464 scope.go:122] "RemoveContainer" containerID="726e8aee3bae429127c2da816464b92aa8ce4e63ab8b60ff798eb89e444e0f18"
Nov 02 12:15:03.490438 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:15:03.490418    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt_odh-ai-gateway-infra(6cae5d40-15f7-4c11-91a2-74d88328fecf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt" podUID="6cae5d40-15f7-4c11-91a2-74d88328fecf"
Nov 02 12:15:03.492246 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.492223    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf" event={"ID":"23be3d03-23fc-418b-8d80-58736eb9217e","Type":"ContainerStarted","Data":"57283c50bed9de35184eafc5d3e6154a97de680456b04b05411db28dc5102209"}
Nov 02 12:15:03.494357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.494336    4464 generic.go:356] "Generic (PLEG): container finished" podID="f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6" containerID="a1b64f94f904a64d9e61000b791e6c365a213bb2440813ad5d839cf0b71c2cbf" exitCode=60
Nov 02 12:15:03.494447 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.494409    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8" event={"ID":"f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6","Type":"ContainerDied","Data":"a1b64f94f904a64d9e61000b791e6c365a213bb2440813ad5d839cf0b71c2cbf"}
Nov 02 12:15:03.494575 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.494560    4464 scope.go:122] "RemoveContainer" containerID="a1b64f94f904a64d9e61000b791e6c365a213bb2440813ad5d839cf0b71c2cbf"
Nov 02 12:15:03.495135 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:15:03.494728    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8_odh-ai-gateway-infra(f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8" podUID="f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6"
Nov 02 12:15:03.512006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.511969    4464 scope.go:122] "RemoveContainer" containerID="1ed16ec831932e7de2bdb097e78f3e88f680625ece4ac77d64abd9793e9ba1ae"
Nov 02 12:15:03.536325 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.536298    4464 scope.go:122] "RemoveContainer" containerID="aa7dbac2638f17a5d12cae2e0b25c4de6d39c776b3250de795b2a5682a5ab70c"
Nov 02 12:15:03.613859 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.613830    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:03.746376 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:03.746331    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf" podStartSLOduration=3.74631706 podStartE2EDuration="3.74631706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:15:03.745923569 +0000 UTC m=+1132.309631979" watchObservedRunningTime="2026-11-02 12:15:03.74631706 +0000 UTC m=+1132.310025468"
Nov 02 12:15:04.125279 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:04.125180    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb" podStartSLOduration=2.125165919 podStartE2EDuration="2.125165919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:15:04.121177182 +0000 UTC m=+1132.684885592" watchObservedRunningTime="2026-11-02 12:15:04.125165919 +0000 UTC m=+1132.688874329"
Nov 02 12:15:04.503305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:04.503272    4464 scope.go:122] "RemoveContainer" containerID="726e8aee3bae429127c2da816464b92aa8ce4e63ab8b60ff798eb89e444e0f18"
Nov 02 12:15:04.503722 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:15:04.503596    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt_odh-ai-gateway-infra(6cae5d40-15f7-4c11-91a2-74d88328fecf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt" podUID="6cae5d40-15f7-4c11-91a2-74d88328fecf"
Nov 02 12:15:04.504840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:04.504813    4464 generic.go:356] "Generic (PLEG): container finished" podID="23be3d03-23fc-418b-8d80-58736eb9217e" containerID="57283c50bed9de35184eafc5d3e6154a97de680456b04b05411db28dc5102209" exitCode=60
Nov 02 12:15:04.504955 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:04.504890    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf" event={"ID":"23be3d03-23fc-418b-8d80-58736eb9217e","Type":"ContainerDied","Data":"57283c50bed9de35184eafc5d3e6154a97de680456b04b05411db28dc5102209"}
Nov 02 12:15:04.504955 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:04.504947    4464 scope.go:122] "RemoveContainer" containerID="d2c6725877202d85dd8585d64897da90b441e1e868ee1156f1854bc7028d95cc"
Nov 02 12:15:04.505125 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:04.505109    4464 scope.go:122] "RemoveContainer" containerID="57283c50bed9de35184eafc5d3e6154a97de680456b04b05411db28dc5102209"
Nov 02 12:15:04.505383 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:15:04.505364    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893695-5hxmf_odh-ai-gateway-infra(23be3d03-23fc-418b-8d80-58736eb9217e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf" podUID="23be3d03-23fc-418b-8d80-58736eb9217e"
Nov 02 12:15:04.506719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:04.506704    4464 scope.go:122] "RemoveContainer" containerID="a1b64f94f904a64d9e61000b791e6c365a213bb2440813ad5d839cf0b71c2cbf"
Nov 02 12:15:04.506962 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:15:04.506940    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8_odh-ai-gateway-infra(f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8" podUID="f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6"
Nov 02 12:15:04.508330 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:04.508312    4464 scope.go:122] "RemoveContainer" containerID="ded45c1e38c042aea8d33bfd73d8905f222d4855d6526c5ead281359a0bc9dd8"
Nov 02 12:15:04.508527 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:15:04.508507    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv_odh-ai-gateway-infra(f99e5052-6768-4b52-ba63-dd93eeae9a99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv" podUID="f99e5052-6768-4b52-ba63-dd93eeae9a99"
Nov 02 12:15:04.510003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:04.509969    4464 generic.go:356] "Generic (PLEG): container finished" podID="176450dd-cb60-4c5c-9f12-110ed3091333" containerID="95b26fb4cbdf5564ef97a977640238ede3849af6c8df6476ba61f9443b2c8233" exitCode=60
Nov 02 12:15:04.510099 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:04.510063    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989" event={"ID":"176450dd-cb60-4c5c-9f12-110ed3091333","Type":"ContainerDied","Data":"95b26fb4cbdf5564ef97a977640238ede3849af6c8df6476ba61f9443b2c8233"}
Nov 02 12:15:04.510221 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:04.510207    4464 scope.go:122] "RemoveContainer" containerID="95b26fb4cbdf5564ef97a977640238ede3849af6c8df6476ba61f9443b2c8233"
Nov 02 12:15:04.510390 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:15:04.510375    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989_odh-ai-gateway-infra(176450dd-cb60-4c5c-9f12-110ed3091333)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989" podUID="176450dd-cb60-4c5c-9f12-110ed3091333"
Nov 02 12:15:04.518383 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:04.518366    4464 scope.go:122] "RemoveContainer" containerID="59d2d52c51099590a4997f995d1d3b7a780299af8c9d52725f734590b2f98d2f"
Nov 02 12:15:04.614986 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:04.614942    4464 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 02 12:15:04.615117 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:04.615005    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb" podUID="4b684afa-613a-43b5-b10d-0f0ece8da04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 02 12:15:05.519597 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:05.519565    4464 scope.go:122] "RemoveContainer" containerID="57283c50bed9de35184eafc5d3e6154a97de680456b04b05411db28dc5102209"
Nov 02 12:15:05.520039 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:15:05.519829    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893695-5hxmf_odh-ai-gateway-infra(23be3d03-23fc-418b-8d80-58736eb9217e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf" podUID="23be3d03-23fc-418b-8d80-58736eb9217e"
Nov 02 12:15:05.521205 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:05.521182    4464 scope.go:122] "RemoveContainer" containerID="95b26fb4cbdf5564ef97a977640238ede3849af6c8df6476ba61f9443b2c8233"
Nov 02 12:15:05.521372 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:15:05.521354    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989_odh-ai-gateway-infra(176450dd-cb60-4c5c-9f12-110ed3091333)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989" podUID="176450dd-cb60-4c5c-9f12-110ed3091333"
Nov 02 12:15:05.613130 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:05.613093    4464 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 02 12:15:05.613281 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:05.613143    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb" podUID="4b684afa-613a-43b5-b10d-0f0ece8da04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 02 12:15:06.613747 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:06.613707    4464 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 02 12:15:06.614155 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:06.613767    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb" podUID="4b684afa-613a-43b5-b10d-0f0ece8da04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 02 12:15:07.612881 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:07.612846    4464 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 02 12:15:07.613044 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:07.612900    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb" podUID="4b684afa-613a-43b5-b10d-0f0ece8da04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 02 12:15:08.613255 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:08.613215    4464 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 02 12:15:08.613637 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:08.613277    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb" podUID="4b684afa-613a-43b5-b10d-0f0ece8da04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 02 12:15:09.616634 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:09.616604    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:09.617126 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:09.617107    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:09.617734 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:09.617716    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:15:11.734864 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:11.734829    4464 scope.go:122] "RemoveContainer" containerID="c5efec9d4b5724f2c515ac355134f30335285eaf45bd70a94af64110e04ef2ca"
Nov 02 12:15:14.581820 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:14.581781    4464 scope.go:122] "RemoveContainer" containerID="a1b64f94f904a64d9e61000b791e6c365a213bb2440813ad5d839cf0b71c2cbf"
Nov 02 12:15:14.582288 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:14.582066    4464 scope.go:122] "RemoveContainer" containerID="57283c50bed9de35184eafc5d3e6154a97de680456b04b05411db28dc5102209"
Nov 02 12:15:15.649051 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:15.639090    4464 scope.go:122] "RemoveContainer" containerID="ded45c1e38c042aea8d33bfd73d8905f222d4855d6526c5ead281359a0bc9dd8"
Nov 02 12:15:15.676100 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:15.676068    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf" event={"ID":"23be3d03-23fc-418b-8d80-58736eb9217e","Type":"ContainerStarted","Data":"6c6aa69b4370404d9acdf26db7247121fbe04038e5104641821c3ab568889aed"}
Nov 02 12:15:15.753001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:15.740244    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8" event={"ID":"f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6","Type":"ContainerStarted","Data":"34ade83280dd9adddaa1adf0dc2084c0ef07c74e1575e24524365a1203d5967c"}
Nov 02 12:15:15.981850 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:15.977237    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8" podStartSLOduration=15.977201378 podStartE2EDuration="15.977201378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:15:15.956626477 +0000 UTC m=+1144.520334888" watchObservedRunningTime="2026-11-02 12:15:15.977201378 +0000 UTC m=+1144.540909788"
Nov 02 12:15:16.578015 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:16.577669    4464 scope.go:122] "RemoveContainer" containerID="726e8aee3bae429127c2da816464b92aa8ce4e63ab8b60ff798eb89e444e0f18"
Nov 02 12:15:16.812002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:16.811096    4464 generic.go:356] "Generic (PLEG): container finished" podID="23be3d03-23fc-418b-8d80-58736eb9217e" containerID="6c6aa69b4370404d9acdf26db7247121fbe04038e5104641821c3ab568889aed" exitCode=60
Nov 02 12:15:16.812002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:16.811227    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf" event={"ID":"23be3d03-23fc-418b-8d80-58736eb9217e","Type":"ContainerDied","Data":"6c6aa69b4370404d9acdf26db7247121fbe04038e5104641821c3ab568889aed"}
Nov 02 12:15:16.812002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:16.811285    4464 scope.go:122] "RemoveContainer" containerID="57283c50bed9de35184eafc5d3e6154a97de680456b04b05411db28dc5102209"
Nov 02 12:15:16.812002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:16.811765    4464 scope.go:122] "RemoveContainer" containerID="6c6aa69b4370404d9acdf26db7247121fbe04038e5104641821c3ab568889aed"
Nov 02 12:15:16.812671 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:15:16.812143    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893695-5hxmf_odh-ai-gateway-infra(23be3d03-23fc-418b-8d80-58736eb9217e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf" podUID="23be3d03-23fc-418b-8d80-58736eb9217e"
Nov 02 12:15:16.869586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:16.868547    4464 generic.go:356] "Generic (PLEG): container finished" podID="f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6" containerID="34ade83280dd9adddaa1adf0dc2084c0ef07c74e1575e24524365a1203d5967c" exitCode=60
Nov 02 12:15:16.869586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:16.868712    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8" event={"ID":"f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6","Type":"ContainerDied","Data":"34ade83280dd9adddaa1adf0dc2084c0ef07c74e1575e24524365a1203d5967c"}
Nov 02 12:15:16.869586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:16.869157    4464 scope.go:122] "RemoveContainer" containerID="34ade83280dd9adddaa1adf0dc2084c0ef07c74e1575e24524365a1203d5967c"
Nov 02 12:15:16.869586 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:15:16.869514    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8_odh-ai-gateway-infra(f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8" podUID="f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6"
Nov 02 12:15:16.881957 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:16.881194    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf" podUID="ff6c0679-4d25-499c-809d-4e22187c23ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:15:16.910080 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:16.910014    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv" event={"ID":"f99e5052-6768-4b52-ba63-dd93eeae9a99","Type":"ContainerStarted","Data":"6fafb3f6e4a1615a4921f8fe9bf188415dd3fabd24201eee50c25bcedc03e590"}
Nov 02 12:15:16.913588 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:16.913565    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf"]
Nov 02 12:15:16.953991 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:16.953709    4464 scope.go:122] "RemoveContainer" containerID="a1b64f94f904a64d9e61000b791e6c365a213bb2440813ad5d839cf0b71c2cbf"
Nov 02 12:15:17.136925 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:17.130414    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8"]
Nov 02 12:15:17.575808 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:17.575771    4464 scope.go:122] "RemoveContainer" containerID="95b26fb4cbdf5564ef97a977640238ede3849af6c8df6476ba61f9443b2c8233"
Nov 02 12:15:17.918917 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:17.918882    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989" event={"ID":"176450dd-cb60-4c5c-9f12-110ed3091333","Type":"ContainerStarted","Data":"b7afba07ef69a0cfd8f218299c3d5b9842f4725b2808ef677645ea019304e164"}
Nov 02 12:15:17.921214 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:17.921191    4464 generic.go:356] "Generic (PLEG): container finished" podID="6cae5d40-15f7-4c11-91a2-74d88328fecf" containerID="087f134dc5ba8b552a06946f61187ddd5274fe89b5095d7a8b93d020827afd5a" exitCode=60
Nov 02 12:15:17.921330 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:17.921264    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt" event={"ID":"6cae5d40-15f7-4c11-91a2-74d88328fecf","Type":"ContainerDied","Data":"087f134dc5ba8b552a06946f61187ddd5274fe89b5095d7a8b93d020827afd5a"}
Nov 02 12:15:17.921393 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:17.921343    4464 scope.go:122] "RemoveContainer" containerID="726e8aee3bae429127c2da816464b92aa8ce4e63ab8b60ff798eb89e444e0f18"
Nov 02 12:15:17.921671 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:17.921651    4464 scope.go:122] "RemoveContainer" containerID="087f134dc5ba8b552a06946f61187ddd5274fe89b5095d7a8b93d020827afd5a"
Nov 02 12:15:17.922021 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:15:17.921959    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt_odh-ai-gateway-infra(6cae5d40-15f7-4c11-91a2-74d88328fecf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt" podUID="6cae5d40-15f7-4c11-91a2-74d88328fecf"
Nov 02 12:15:17.926786 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:17.926766    4464 generic.go:356] "Generic (PLEG): container finished" podID="f99e5052-6768-4b52-ba63-dd93eeae9a99" containerID="6fafb3f6e4a1615a4921f8fe9bf188415dd3fabd24201eee50c25bcedc03e590" exitCode=60
Nov 02 12:15:17.926888 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:17.926803    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv" event={"ID":"f99e5052-6768-4b52-ba63-dd93eeae9a99","Type":"ContainerDied","Data":"6fafb3f6e4a1615a4921f8fe9bf188415dd3fabd24201eee50c25bcedc03e590"}
Nov 02 12:15:17.927095 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:17.927081    4464 scope.go:122] "RemoveContainer" containerID="6fafb3f6e4a1615a4921f8fe9bf188415dd3fabd24201eee50c25bcedc03e590"
Nov 02 12:15:17.927344 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:15:17.927326    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv_odh-ai-gateway-infra(f99e5052-6768-4b52-ba63-dd93eeae9a99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv" podUID="f99e5052-6768-4b52-ba63-dd93eeae9a99"
Nov 02 12:15:17.935117 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:17.935097    4464 scope.go:122] "RemoveContainer" containerID="ded45c1e38c042aea8d33bfd73d8905f222d4855d6526c5ead281359a0bc9dd8"
Nov 02 12:15:18.086635 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.086605    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf"
Nov 02 12:15:18.118867 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.118845    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8"
Nov 02 12:15:18.141161 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.141137    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23be3d03-23fc-418b-8d80-58736eb9217e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23be3d03-23fc-418b-8d80-58736eb9217e-maas-api-service-ca\") pod \"23be3d03-23fc-418b-8d80-58736eb9217e\" (UID: \"23be3d03-23fc-418b-8d80-58736eb9217e\") "
Nov 02 12:15:18.141299 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.141193    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23be3d03-23fc-418b-8d80-58736eb9217e-kube-api-access-xkhft\" (UniqueName: \"kubernetes.io/projected/23be3d03-23fc-418b-8d80-58736eb9217e-kube-api-access-xkhft\") pod \"23be3d03-23fc-418b-8d80-58736eb9217e\" (UID: \"23be3d03-23fc-418b-8d80-58736eb9217e\") "
Nov 02 12:15:18.144002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.142690    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23be3d03-23fc-418b-8d80-58736eb9217e-maas-api-service-ca" pod "23be3d03-23fc-418b-8d80-58736eb9217e" (UID: "23be3d03-23fc-418b-8d80-58736eb9217e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:15:18.161644 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.161606    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23be3d03-23fc-418b-8d80-58736eb9217e-kube-api-access-xkhft" pod "23be3d03-23fc-418b-8d80-58736eb9217e" (UID: "23be3d03-23fc-418b-8d80-58736eb9217e"). InnerVolumeSpecName "kube-api-access-xkhft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:15:18.197402 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.197365    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv"]
Nov 02 12:15:18.243768 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.243735    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6-maas-api-service-ca\") pod \"f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6\" (UID: \"f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6\") "
Nov 02 12:15:18.243963 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.243844    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6-kube-api-access-n7j67\" (UniqueName: \"kubernetes.io/projected/f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6-kube-api-access-n7j67\") pod \"f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6\" (UID: \"f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6\") "
Nov 02 12:15:18.244193 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.244170    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6-maas-api-service-ca" pod "f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6" (UID: "f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:15:18.244455 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.244433    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23be3d03-23fc-418b-8d80-58736eb9217e-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:18.244544 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.244460    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkhft\" (UniqueName: \"kubernetes.io/projected/23be3d03-23fc-418b-8d80-58736eb9217e-kube-api-access-xkhft\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:18.244544 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.244478    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:18.246294 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.246261    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6-kube-api-access-n7j67" pod "f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6" (UID: "f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6"). InnerVolumeSpecName "kube-api-access-n7j67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:15:18.346878 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.346810    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7j67\" (UniqueName: \"kubernetes.io/projected/f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6-kube-api-access-n7j67\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:18.597901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.597870    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"]
Nov 02 12:15:18.598462 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.598449    4464 cpu_manager.go:414] "containerMap: removing container" podUID="23be3d03-23fc-418b-8d80-58736eb9217e" containerName="cleanup"
Nov 02 12:15:18.598526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.598463    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23be3d03-23fc-418b-8d80-58736eb9217e" containerName="cleanup"
Nov 02 12:15:18.598526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.598479    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6" containerName="cleanup"
Nov 02 12:15:18.598526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.598484    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6" containerName="cleanup"
Nov 02 12:15:18.598526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.598490    4464 cpu_manager.go:414] "containerMap: removing container" podUID="23be3d03-23fc-418b-8d80-58736eb9217e" containerName="cleanup"
Nov 02 12:15:18.598526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.598495    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23be3d03-23fc-418b-8d80-58736eb9217e" containerName="cleanup"
Nov 02 12:15:18.598526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.598517    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6" containerName="cleanup"
Nov 02 12:15:18.598526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.598521    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6" containerName="cleanup"
Nov 02 12:15:18.598741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.598625    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6" containerName="cleanup"
Nov 02 12:15:18.598741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.598639    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="23be3d03-23fc-418b-8d80-58736eb9217e" containerName="cleanup"
Nov 02 12:15:18.598741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.598646    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="23be3d03-23fc-418b-8d80-58736eb9217e" containerName="cleanup"
Nov 02 12:15:18.598741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.598652    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6" containerName="cleanup"
Nov 02 12:15:18.602020 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.602004    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"
Nov 02 12:15:18.651212 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.651130    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202b9e62-26ac-4c0b-b479-e9469ba7e547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e23f5b7-blz8x\" (UID: \"202b9e62-26ac-4c0b-b479-e9469ba7e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"
Nov 02 12:15:18.651212 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.651182    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202b9e62-26ac-4c0b-b479-e9469ba7e547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e23f5b7-blz8x\" (UID: \"202b9e62-26ac-4c0b-b479-e9469ba7e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"
Nov 02 12:15:18.651434 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.651267    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2qzb\" (UniqueName: \"kubernetes.io/projected/202b9e62-26ac-4c0b-b479-e9469ba7e547-kube-api-access-n2qzb\") pod \"maas-api-revoke-sub-7e23f5b7-blz8x\" (UID: \"202b9e62-26ac-4c0b-b479-e9469ba7e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"
Nov 02 12:15:18.657806 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.657770    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:15:18.662729 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.662705    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"]
Nov 02 12:15:18.753742 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.753305    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202b9e62-26ac-4c0b-b479-e9469ba7e547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e23f5b7-blz8x\" (UID: \"202b9e62-26ac-4c0b-b479-e9469ba7e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"
Nov 02 12:15:18.753742 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.753377    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202b9e62-26ac-4c0b-b479-e9469ba7e547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e23f5b7-blz8x\" (UID: \"202b9e62-26ac-4c0b-b479-e9469ba7e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"
Nov 02 12:15:18.753742 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.753410    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2qzb\" (UniqueName: \"kubernetes.io/projected/202b9e62-26ac-4c0b-b479-e9469ba7e547-kube-api-access-n2qzb\") pod \"maas-api-revoke-sub-7e23f5b7-blz8x\" (UID: \"202b9e62-26ac-4c0b-b479-e9469ba7e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"
Nov 02 12:15:18.754271 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.754226    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202b9e62-26ac-4c0b-b479-e9469ba7e547-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e23f5b7-blz8x\" (UID: \"202b9e62-26ac-4c0b-b479-e9469ba7e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"
Nov 02 12:15:18.756395 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.756346    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202b9e62-26ac-4c0b-b479-e9469ba7e547-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e23f5b7-blz8x\" (UID: \"202b9e62-26ac-4c0b-b479-e9469ba7e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"
Nov 02 12:15:18.814935 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.814908    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2qzb\" (UniqueName: \"kubernetes.io/projected/202b9e62-26ac-4c0b-b479-e9469ba7e547-kube-api-access-n2qzb\") pod \"maas-api-revoke-sub-7e23f5b7-blz8x\" (UID: \"202b9e62-26ac-4c0b-b479-e9469ba7e547\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"
Nov 02 12:15:18.916259 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.916052    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"
Nov 02 12:15:18.937551 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.937521    4464 generic.go:356] "Generic (PLEG): container finished" podID="176450dd-cb60-4c5c-9f12-110ed3091333" containerID="b7afba07ef69a0cfd8f218299c3d5b9842f4725b2808ef677645ea019304e164" exitCode=60
Nov 02 12:15:18.937961 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.937594    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989" event={"ID":"176450dd-cb60-4c5c-9f12-110ed3091333","Type":"ContainerDied","Data":"b7afba07ef69a0cfd8f218299c3d5b9842f4725b2808ef677645ea019304e164"}
Nov 02 12:15:18.937961 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.937655    4464 scope.go:122] "RemoveContainer" containerID="95b26fb4cbdf5564ef97a977640238ede3849af6c8df6476ba61f9443b2c8233"
Nov 02 12:15:18.938126 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.938082    4464 scope.go:122] "RemoveContainer" containerID="b7afba07ef69a0cfd8f218299c3d5b9842f4725b2808ef677645ea019304e164"
Nov 02 12:15:18.938449 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:15:18.938412    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989_odh-ai-gateway-infra(176450dd-cb60-4c5c-9f12-110ed3091333)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989" podUID="176450dd-cb60-4c5c-9f12-110ed3091333"
Nov 02 12:15:18.941127 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.941099    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf"
Nov 02 12:15:18.941127 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.941109    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf" event={"ID":"23be3d03-23fc-418b-8d80-58736eb9217e","Type":"ContainerDied","Data":"a0790367ff8c05e87655b1aae2b29a179f8552834d5573349361ed6b7f710aa3"}
Nov 02 12:15:18.943047 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.943023    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8" event={"ID":"f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6","Type":"ContainerDied","Data":"fdd347ffdf907e17f309c0b7afa11d9f974da45455b0caf3e95b00413a46334d"}
Nov 02 12:15:18.943287 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.943031    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8"
Nov 02 12:15:18.962174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.962145    4464 scope.go:122] "RemoveContainer" containerID="6c6aa69b4370404d9acdf26db7247121fbe04038e5104641821c3ab568889aed"
Nov 02 12:15:18.974392 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:18.974372    4464 scope.go:122] "RemoveContainer" containerID="34ade83280dd9adddaa1adf0dc2084c0ef07c74e1575e24524365a1203d5967c"
Nov 02 12:15:19.060735 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.060690    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989"]
Nov 02 12:15:19.118728 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.118703    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv"
Nov 02 12:15:19.179104 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.179037    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf"]
Nov 02 12:15:19.183494 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.183465    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893695-5hxmf"]
Nov 02 12:15:19.264167 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.264132    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f99e5052-6768-4b52-ba63-dd93eeae9a99-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f99e5052-6768-4b52-ba63-dd93eeae9a99-maas-api-service-ca\") pod \"f99e5052-6768-4b52-ba63-dd93eeae9a99\" (UID: \"f99e5052-6768-4b52-ba63-dd93eeae9a99\") "
Nov 02 12:15:19.264330 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.264274    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99e5052-6768-4b52-ba63-dd93eeae9a99-kube-api-access-wbfjn\" (UniqueName: \"kubernetes.io/projected/f99e5052-6768-4b52-ba63-dd93eeae9a99-kube-api-access-wbfjn\") pod \"f99e5052-6768-4b52-ba63-dd93eeae9a99\" (UID: \"f99e5052-6768-4b52-ba63-dd93eeae9a99\") "
Nov 02 12:15:19.264527 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.264504    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99e5052-6768-4b52-ba63-dd93eeae9a99-maas-api-service-ca" pod "f99e5052-6768-4b52-ba63-dd93eeae9a99" (UID: "f99e5052-6768-4b52-ba63-dd93eeae9a99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:15:19.264693 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.264676    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f99e5052-6768-4b52-ba63-dd93eeae9a99-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:19.266294 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.266268    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99e5052-6768-4b52-ba63-dd93eeae9a99-kube-api-access-wbfjn" pod "f99e5052-6768-4b52-ba63-dd93eeae9a99" (UID: "f99e5052-6768-4b52-ba63-dd93eeae9a99"). InnerVolumeSpecName "kube-api-access-wbfjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:15:19.306155 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.306119    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"]
Nov 02 12:15:19.366076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.366053    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbfjn\" (UniqueName: \"kubernetes.io/projected/f99e5052-6768-4b52-ba63-dd93eeae9a99-kube-api-access-wbfjn\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:19.411430 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.411392    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8"]
Nov 02 12:15:19.419294 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.419259    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893695-q4hw8"]
Nov 02 12:15:19.525578 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.525546    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt"]
Nov 02 12:15:19.583415 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.583379    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23be3d03-23fc-418b-8d80-58736eb9217e" path="/var/lib/kubelet/pods/23be3d03-23fc-418b-8d80-58736eb9217e/volumes"
Nov 02 12:15:19.583903 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.583879    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6" path="/var/lib/kubelet/pods/f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6/volumes"
Nov 02 12:15:19.673381 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.673354    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt"
Nov 02 12:15:19.771307 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.771228    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cae5d40-15f7-4c11-91a2-74d88328fecf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cae5d40-15f7-4c11-91a2-74d88328fecf-maas-api-service-ca\") pod \"6cae5d40-15f7-4c11-91a2-74d88328fecf\" (UID: \"6cae5d40-15f7-4c11-91a2-74d88328fecf\") "
Nov 02 12:15:19.771307 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.771282    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cae5d40-15f7-4c11-91a2-74d88328fecf-kube-api-access-kkhrm\" (UniqueName: \"kubernetes.io/projected/6cae5d40-15f7-4c11-91a2-74d88328fecf-kube-api-access-kkhrm\") pod \"6cae5d40-15f7-4c11-91a2-74d88328fecf\" (UID: \"6cae5d40-15f7-4c11-91a2-74d88328fecf\") "
Nov 02 12:15:19.771652 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.771624    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cae5d40-15f7-4c11-91a2-74d88328fecf-maas-api-service-ca" pod "6cae5d40-15f7-4c11-91a2-74d88328fecf" (UID: "6cae5d40-15f7-4c11-91a2-74d88328fecf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:15:19.771904 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.771887    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cae5d40-15f7-4c11-91a2-74d88328fecf-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:19.773350 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.773326    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cae5d40-15f7-4c11-91a2-74d88328fecf-kube-api-access-kkhrm" pod "6cae5d40-15f7-4c11-91a2-74d88328fecf" (UID: "6cae5d40-15f7-4c11-91a2-74d88328fecf"). InnerVolumeSpecName "kube-api-access-kkhrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:15:19.873080 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.873051    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkhrm\" (UniqueName: \"kubernetes.io/projected/6cae5d40-15f7-4c11-91a2-74d88328fecf-kube-api-access-kkhrm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:19.951637 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.951606    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt"
Nov 02 12:15:19.951637 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.951626    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt" event={"ID":"6cae5d40-15f7-4c11-91a2-74d88328fecf","Type":"ContainerDied","Data":"11661a60a32403bb1c7d2deee60ceb8cacbf5ca8ede90d403cf175d93ac24528"}
Nov 02 12:15:19.952183 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.951690    4464 scope.go:122] "RemoveContainer" containerID="087f134dc5ba8b552a06946f61187ddd5274fe89b5095d7a8b93d020827afd5a"
Nov 02 12:15:19.969445 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.969408    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x" event={"ID":"202b9e62-26ac-4c0b-b479-e9469ba7e547","Type":"ContainerStarted","Data":"512d6d81a5938e5d5f01e4a16cf237d2572519e616226ef765b6853479c1cfe7"}
Nov 02 12:15:19.969561 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.969514    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x" event={"ID":"202b9e62-26ac-4c0b-b479-e9469ba7e547","Type":"ContainerStarted","Data":"ec93570aad163c8acd552a867b96e8099ac7c6efab1f0ce25975498efde4ff6a"}
Nov 02 12:15:19.971040 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.971016    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv" event={"ID":"f99e5052-6768-4b52-ba63-dd93eeae9a99","Type":"ContainerDied","Data":"ca3ffc0416ca7a69a1272943858512e6df723f5c07aa002770373203d6f9f368"}
Nov 02 12:15:19.971150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.971024    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv"
Nov 02 12:15:19.971150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:19.971071    4464 scope.go:122] "RemoveContainer" containerID="6fafb3f6e4a1615a4921f8fe9bf188415dd3fabd24201eee50c25bcedc03e590"
Nov 02 12:15:20.113605 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:20.113578    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989"
Nov 02 12:15:20.176031 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:20.176004    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/176450dd-cb60-4c5c-9f12-110ed3091333-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/176450dd-cb60-4c5c-9f12-110ed3091333-maas-api-service-ca\") pod \"176450dd-cb60-4c5c-9f12-110ed3091333\" (UID: \"176450dd-cb60-4c5c-9f12-110ed3091333\") "
Nov 02 12:15:20.176174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:20.176061    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176450dd-cb60-4c5c-9f12-110ed3091333-kube-api-access-hdpkt\" (UniqueName: \"kubernetes.io/projected/176450dd-cb60-4c5c-9f12-110ed3091333-kube-api-access-hdpkt\") pod \"176450dd-cb60-4c5c-9f12-110ed3091333\" (UID: \"176450dd-cb60-4c5c-9f12-110ed3091333\") "
Nov 02 12:15:20.176384 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:20.176352    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176450dd-cb60-4c5c-9f12-110ed3091333-maas-api-service-ca" pod "176450dd-cb60-4c5c-9f12-110ed3091333" (UID: "176450dd-cb60-4c5c-9f12-110ed3091333"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:15:20.176566 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:20.176552    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/176450dd-cb60-4c5c-9f12-110ed3091333-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:20.178093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:20.178070    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176450dd-cb60-4c5c-9f12-110ed3091333-kube-api-access-hdpkt" pod "176450dd-cb60-4c5c-9f12-110ed3091333" (UID: "176450dd-cb60-4c5c-9f12-110ed3091333"). InnerVolumeSpecName "kube-api-access-hdpkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:15:20.248924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:20.248875    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x" podStartSLOduration=2.248860902 podStartE2EDuration="2.248860902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:15:20.247096521 +0000 UTC m=+1148.810804952" watchObservedRunningTime="2026-11-02 12:15:20.248860902 +0000 UTC m=+1148.812569312"
Nov 02 12:15:20.278149 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:20.278124    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdpkt\" (UniqueName: \"kubernetes.io/projected/176450dd-cb60-4c5c-9f12-110ed3091333-kube-api-access-hdpkt\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:20.362005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:20.361894    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt"]
Nov 02 12:15:20.366768 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:20.366737    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893695-8nkpt"]
Nov 02 12:15:20.477961 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:20.477928    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv"]
Nov 02 12:15:20.482906 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:20.482879    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-036f88-29893695-jkzvv"]
Nov 02 12:15:20.986689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:20.986659    4464 generic.go:356] "Generic (PLEG): container finished" podID="202b9e62-26ac-4c0b-b479-e9469ba7e547" containerID="512d6d81a5938e5d5f01e4a16cf237d2572519e616226ef765b6853479c1cfe7" exitCode=0
Nov 02 12:15:20.987208 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:20.986757    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x" event={"ID":"202b9e62-26ac-4c0b-b479-e9469ba7e547","Type":"ContainerDied","Data":"512d6d81a5938e5d5f01e4a16cf237d2572519e616226ef765b6853479c1cfe7"}
Nov 02 12:15:20.989535 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:20.989511    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989"
Nov 02 12:15:20.989631 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:20.989544    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989" event={"ID":"176450dd-cb60-4c5c-9f12-110ed3091333","Type":"ContainerDied","Data":"593152a548d3a0b142424f4e60d4ea2b0315c71fa21bd7c095552b4397e387d3"}
Nov 02 12:15:20.989695 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:20.989634    4464 scope.go:122] "RemoveContainer" containerID="b7afba07ef69a0cfd8f218299c3d5b9842f4725b2808ef677645ea019304e164"
Nov 02 12:15:21.223101 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:21.223072    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989"]
Nov 02 12:15:21.233383 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:21.233359    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893695-25989"]
Nov 02 12:15:21.582915 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:21.582882    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176450dd-cb60-4c5c-9f12-110ed3091333" path="/var/lib/kubelet/pods/176450dd-cb60-4c5c-9f12-110ed3091333/volumes"
Nov 02 12:15:21.583325 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:21.583309    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cae5d40-15f7-4c11-91a2-74d88328fecf" path="/var/lib/kubelet/pods/6cae5d40-15f7-4c11-91a2-74d88328fecf/volumes"
Nov 02 12:15:21.583734 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:21.583719    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f99e5052-6768-4b52-ba63-dd93eeae9a99" path="/var/lib/kubelet/pods/f99e5052-6768-4b52-ba63-dd93eeae9a99/volumes"
Nov 02 12:15:22.150729 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:22.150707    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"
Nov 02 12:15:22.299584 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:22.299494    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/202b9e62-26ac-4c0b-b479-e9469ba7e547-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202b9e62-26ac-4c0b-b479-e9469ba7e547-maas-api-xks-ca\") pod \"202b9e62-26ac-4c0b-b479-e9469ba7e547\" (UID: \"202b9e62-26ac-4c0b-b479-e9469ba7e547\") "
Nov 02 12:15:22.299769 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:22.299747    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202b9e62-26ac-4c0b-b479-e9469ba7e547-kube-api-access-n2qzb\" (UniqueName: \"kubernetes.io/projected/202b9e62-26ac-4c0b-b479-e9469ba7e547-kube-api-access-n2qzb\") pod \"202b9e62-26ac-4c0b-b479-e9469ba7e547\" (UID: \"202b9e62-26ac-4c0b-b479-e9469ba7e547\") "
Nov 02 12:15:22.299853 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:22.299820    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202b9e62-26ac-4c0b-b479-e9469ba7e547-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202b9e62-26ac-4c0b-b479-e9469ba7e547-maas-api-openshift-ca\") pod \"202b9e62-26ac-4c0b-b479-e9469ba7e547\" (UID: \"202b9e62-26ac-4c0b-b479-e9469ba7e547\") "
Nov 02 12:15:22.300263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:22.300234    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202b9e62-26ac-4c0b-b479-e9469ba7e547-maas-api-openshift-ca" pod "202b9e62-26ac-4c0b-b479-e9469ba7e547" (UID: "202b9e62-26ac-4c0b-b479-e9469ba7e547"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:15:22.300381 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:22.300347    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/202b9e62-26ac-4c0b-b479-e9469ba7e547-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:22.301746 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:22.301720    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202b9e62-26ac-4c0b-b479-e9469ba7e547-maas-api-xks-ca" pod "202b9e62-26ac-4c0b-b479-e9469ba7e547" (UID: "202b9e62-26ac-4c0b-b479-e9469ba7e547"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:15:22.302421 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:22.302398    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202b9e62-26ac-4c0b-b479-e9469ba7e547-kube-api-access-n2qzb" pod "202b9e62-26ac-4c0b-b479-e9469ba7e547" (UID: "202b9e62-26ac-4c0b-b479-e9469ba7e547"). InnerVolumeSpecName "kube-api-access-n2qzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:15:22.401238 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:22.401212    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2qzb\" (UniqueName: \"kubernetes.io/projected/202b9e62-26ac-4c0b-b479-e9469ba7e547-kube-api-access-n2qzb\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:22.401238 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:22.401238    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/202b9e62-26ac-4c0b-b479-e9469ba7e547-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:23.008644 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.008610    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"
Nov 02 12:15:23.008831 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.008648    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x" event={"ID":"202b9e62-26ac-4c0b-b479-e9469ba7e547","Type":"ContainerDied","Data":"ec93570aad163c8acd552a867b96e8099ac7c6efab1f0ce25975498efde4ff6a"}
Nov 02 12:15:23.008831 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.008724    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec93570aad163c8acd552a867b96e8099ac7c6efab1f0ce25975498efde4ff6a"
Nov 02 12:15:23.830347 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.830316    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"]
Nov 02 12:15:23.831020 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831003    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6" containerName="cleanup"
Nov 02 12:15:23.831076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831022    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6" containerName="cleanup"
Nov 02 12:15:23.831076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831040    4464 cpu_manager.go:414] "containerMap: removing container" podUID="176450dd-cb60-4c5c-9f12-110ed3091333" containerName="cleanup"
Nov 02 12:15:23.831076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831047    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176450dd-cb60-4c5c-9f12-110ed3091333" containerName="cleanup"
Nov 02 12:15:23.831076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831060    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f99e5052-6768-4b52-ba63-dd93eeae9a99" containerName="cleanup"
Nov 02 12:15:23.831076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831066    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99e5052-6768-4b52-ba63-dd93eeae9a99" containerName="cleanup"
Nov 02 12:15:23.831076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831073    4464 cpu_manager.go:414] "containerMap: removing container" podUID="202b9e62-26ac-4c0b-b479-e9469ba7e547" containerName="revoke-keys"
Nov 02 12:15:23.831076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831078    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202b9e62-26ac-4c0b-b479-e9469ba7e547" containerName="revoke-keys"
Nov 02 12:15:23.831278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831092    4464 cpu_manager.go:414] "containerMap: removing container" podUID="176450dd-cb60-4c5c-9f12-110ed3091333" containerName="cleanup"
Nov 02 12:15:23.831278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831097    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176450dd-cb60-4c5c-9f12-110ed3091333" containerName="cleanup"
Nov 02 12:15:23.831278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831111    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6cae5d40-15f7-4c11-91a2-74d88328fecf" containerName="cleanup"
Nov 02 12:15:23.831278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831116    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cae5d40-15f7-4c11-91a2-74d88328fecf" containerName="cleanup"
Nov 02 12:15:23.831278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831125    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f99e5052-6768-4b52-ba63-dd93eeae9a99" containerName="cleanup"
Nov 02 12:15:23.831278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831129    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99e5052-6768-4b52-ba63-dd93eeae9a99" containerName="cleanup"
Nov 02 12:15:23.831278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831134    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6cae5d40-15f7-4c11-91a2-74d88328fecf" containerName="cleanup"
Nov 02 12:15:23.831278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831139    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cae5d40-15f7-4c11-91a2-74d88328fecf" containerName="cleanup"
Nov 02 12:15:23.831278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831147    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6cae5d40-15f7-4c11-91a2-74d88328fecf" containerName="cleanup"
Nov 02 12:15:23.831278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831152    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cae5d40-15f7-4c11-91a2-74d88328fecf" containerName="cleanup"
Nov 02 12:15:23.831278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831164    4464 cpu_manager.go:414] "containerMap: removing container" podUID="23be3d03-23fc-418b-8d80-58736eb9217e" containerName="cleanup"
Nov 02 12:15:23.831278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831171    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23be3d03-23fc-418b-8d80-58736eb9217e" containerName="cleanup"
Nov 02 12:15:23.831615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831302    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="176450dd-cb60-4c5c-9f12-110ed3091333" containerName="cleanup"
Nov 02 12:15:23.831615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831317    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99e5052-6768-4b52-ba63-dd93eeae9a99" containerName="cleanup"
Nov 02 12:15:23.831615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831327    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cae5d40-15f7-4c11-91a2-74d88328fecf" containerName="cleanup"
Nov 02 12:15:23.831615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831333    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99e5052-6768-4b52-ba63-dd93eeae9a99" containerName="cleanup"
Nov 02 12:15:23.831615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831339    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cae5d40-15f7-4c11-91a2-74d88328fecf" containerName="cleanup"
Nov 02 12:15:23.831615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831346    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="23be3d03-23fc-418b-8d80-58736eb9217e" containerName="cleanup"
Nov 02 12:15:23.831615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831351    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cae5d40-15f7-4c11-91a2-74d88328fecf" containerName="cleanup"
Nov 02 12:15:23.831615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831357    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43c5b36-e9bb-4646-bec4-e2ccf98b7bb6" containerName="cleanup"
Nov 02 12:15:23.831615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831368    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="202b9e62-26ac-4c0b-b479-e9469ba7e547" containerName="revoke-keys"
Nov 02 12:15:23.831615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.831375    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="176450dd-cb60-4c5c-9f12-110ed3091333" containerName="cleanup"
Nov 02 12:15:23.835042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.835026    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"
Nov 02 12:15:23.892205 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.892172    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"]
Nov 02 12:15:23.918370 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.918339    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5w4q\" (UniqueName: \"kubernetes.io/projected/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-kube-api-access-p5w4q\") pod \"maas-api-revoke-sub-bf897492-bflt7\" (UID: \"43fdfd1b-9092-4d87-8bff-e84eb05caaf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"
Nov 02 12:15:23.918525 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.918472    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf897492-bflt7\" (UID: \"43fdfd1b-9092-4d87-8bff-e84eb05caaf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"
Nov 02 12:15:23.918567 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.918537    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf897492-bflt7\" (UID: \"43fdfd1b-9092-4d87-8bff-e84eb05caaf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"
Nov 02 12:15:23.937791 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.937768    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:15:23.938074 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.938055    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:15:23.938129 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.938081    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:15:23.939214 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:23.939196    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:15:24.020194 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:24.020161    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf897492-bflt7\" (UID: \"43fdfd1b-9092-4d87-8bff-e84eb05caaf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"
Nov 02 12:15:24.020344 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:24.020252    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf897492-bflt7\" (UID: \"43fdfd1b-9092-4d87-8bff-e84eb05caaf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"
Nov 02 12:15:24.020344 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:24.020323    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5w4q\" (UniqueName: \"kubernetes.io/projected/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-kube-api-access-p5w4q\") pod \"maas-api-revoke-sub-bf897492-bflt7\" (UID: \"43fdfd1b-9092-4d87-8bff-e84eb05caaf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"
Nov 02 12:15:24.020945 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:24.020922    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf897492-bflt7\" (UID: \"43fdfd1b-9092-4d87-8bff-e84eb05caaf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"
Nov 02 12:15:24.022497 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:24.022478    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf897492-bflt7\" (UID: \"43fdfd1b-9092-4d87-8bff-e84eb05caaf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"
Nov 02 12:15:24.082434 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:24.082369    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5w4q\" (UniqueName: \"kubernetes.io/projected/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-kube-api-access-p5w4q\") pod \"maas-api-revoke-sub-bf897492-bflt7\" (UID: \"43fdfd1b-9092-4d87-8bff-e84eb05caaf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"
Nov 02 12:15:24.149077 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:24.149052    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"
Nov 02 12:15:24.528635 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:15:24.528598    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43fdfd1b_9092_4d87_8bff_e84eb05caaf0.slice/crio-186a967675fe5c387df3f11f92d27eb0eb88bd13e08a6dcf746c4bc60bec7e5d WatchSource:0}: Error finding container 186a967675fe5c387df3f11f92d27eb0eb88bd13e08a6dcf746c4bc60bec7e5d: Status 404 returned error can't find the container with id 186a967675fe5c387df3f11f92d27eb0eb88bd13e08a6dcf746c4bc60bec7e5d
Nov 02 12:15:24.529581 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:24.529556    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"]
Nov 02 12:15:25.023563 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:25.023524    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7" event={"ID":"43fdfd1b-9092-4d87-8bff-e84eb05caaf0","Type":"ContainerStarted","Data":"aee803f7766dac33c3bdc57e6dbe5b463042bfdebd45476b916d6b8fdfc1a577"}
Nov 02 12:15:25.023962 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:25.023594    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7" event={"ID":"43fdfd1b-9092-4d87-8bff-e84eb05caaf0","Type":"ContainerStarted","Data":"186a967675fe5c387df3f11f92d27eb0eb88bd13e08a6dcf746c4bc60bec7e5d"}
Nov 02 12:15:25.148329 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:25.148222    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7" podStartSLOduration=2.148199725 podStartE2EDuration="2.148199725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:15:25.146796718 +0000 UTC m=+1153.710505144" watchObservedRunningTime="2026-11-02 12:15:25.148199725 +0000 UTC m=+1153.711908135"
Nov 02 12:15:26.045508 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:26.045463    4464 generic.go:356] "Generic (PLEG): container finished" podID="43fdfd1b-9092-4d87-8bff-e84eb05caaf0" containerID="aee803f7766dac33c3bdc57e6dbe5b463042bfdebd45476b916d6b8fdfc1a577" exitCode=0
Nov 02 12:15:26.045939 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:26.045538    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7" event={"ID":"43fdfd1b-9092-4d87-8bff-e84eb05caaf0","Type":"ContainerDied","Data":"aee803f7766dac33c3bdc57e6dbe5b463042bfdebd45476b916d6b8fdfc1a577"}
Nov 02 12:15:27.186029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:27.186006    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"
Nov 02 12:15:27.267010 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:27.266962    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-maas-api-openshift-ca\") pod \"43fdfd1b-9092-4d87-8bff-e84eb05caaf0\" (UID: \"43fdfd1b-9092-4d87-8bff-e84eb05caaf0\") "
Nov 02 12:15:27.267132 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:27.267071    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-maas-api-xks-ca\") pod \"43fdfd1b-9092-4d87-8bff-e84eb05caaf0\" (UID: \"43fdfd1b-9092-4d87-8bff-e84eb05caaf0\") "
Nov 02 12:15:27.267190 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:27.267173    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-kube-api-access-p5w4q\" (UniqueName: \"kubernetes.io/projected/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-kube-api-access-p5w4q\") pod \"43fdfd1b-9092-4d87-8bff-e84eb05caaf0\" (UID: \"43fdfd1b-9092-4d87-8bff-e84eb05caaf0\") "
Nov 02 12:15:27.267283 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:27.267257    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-maas-api-openshift-ca" pod "43fdfd1b-9092-4d87-8bff-e84eb05caaf0" (UID: "43fdfd1b-9092-4d87-8bff-e84eb05caaf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:15:27.267602 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:27.267584    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:27.269111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:27.269090    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-maas-api-xks-ca" pod "43fdfd1b-9092-4d87-8bff-e84eb05caaf0" (UID: "43fdfd1b-9092-4d87-8bff-e84eb05caaf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:15:27.269277 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:27.269262    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-kube-api-access-p5w4q" pod "43fdfd1b-9092-4d87-8bff-e84eb05caaf0" (UID: "43fdfd1b-9092-4d87-8bff-e84eb05caaf0"). InnerVolumeSpecName "kube-api-access-p5w4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:15:27.369398 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:27.369317    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:27.369398 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:27.369348    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5w4q\" (UniqueName: \"kubernetes.io/projected/43fdfd1b-9092-4d87-8bff-e84eb05caaf0-kube-api-access-p5w4q\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:28.031037 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:28.031001    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:15:28.059515 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:28.059490    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"
Nov 02 12:15:28.059515 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:28.059503    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7" event={"ID":"43fdfd1b-9092-4d87-8bff-e84eb05caaf0","Type":"ContainerDied","Data":"186a967675fe5c387df3f11f92d27eb0eb88bd13e08a6dcf746c4bc60bec7e5d"}
Nov 02 12:15:28.059683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:28.059542    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="186a967675fe5c387df3f11f92d27eb0eb88bd13e08a6dcf746c4bc60bec7e5d"
Nov 02 12:15:28.411289 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:15:28.411186    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:15:34.508681 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.508647    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"]
Nov 02 12:15:34.509248 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.509234    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f99e5052-6768-4b52-ba63-dd93eeae9a99" containerName="cleanup"
Nov 02 12:15:34.509305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.509249    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99e5052-6768-4b52-ba63-dd93eeae9a99" containerName="cleanup"
Nov 02 12:15:34.509305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.509266    4464 cpu_manager.go:414] "containerMap: removing container" podUID="176450dd-cb60-4c5c-9f12-110ed3091333" containerName="cleanup"
Nov 02 12:15:34.509305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.509270    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176450dd-cb60-4c5c-9f12-110ed3091333" containerName="cleanup"
Nov 02 12:15:34.509305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.509299    4464 cpu_manager.go:414] "containerMap: removing container" podUID="43fdfd1b-9092-4d87-8bff-e84eb05caaf0" containerName="revoke-keys"
Nov 02 12:15:34.509305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.509305    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43fdfd1b-9092-4d87-8bff-e84eb05caaf0" containerName="revoke-keys"
Nov 02 12:15:34.509459 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.509404    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="43fdfd1b-9092-4d87-8bff-e84eb05caaf0" containerName="revoke-keys"
Nov 02 12:15:34.509459 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.509423    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="176450dd-cb60-4c5c-9f12-110ed3091333" containerName="cleanup"
Nov 02 12:15:34.509459 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.509429    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99e5052-6768-4b52-ba63-dd93eeae9a99" containerName="cleanup"
Nov 02 12:15:34.513095 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.513076    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"
Nov 02 12:15:34.571296 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.571267    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"]
Nov 02 12:15:34.587935 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.587909    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkz98\" (UniqueName: \"kubernetes.io/projected/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-kube-api-access-vkz98\") pod \"maas-api-revoke-sub-57057b37-6rgvn\" (UID: \"dfbd370f-d4f3-440b-aa8d-e0d86efb9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"
Nov 02 12:15:34.588193 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.588168    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57057b37-6rgvn\" (UID: \"dfbd370f-d4f3-440b-aa8d-e0d86efb9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"
Nov 02 12:15:34.588255 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.588215    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57057b37-6rgvn\" (UID: \"dfbd370f-d4f3-440b-aa8d-e0d86efb9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"
Nov 02 12:15:34.616570 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.616550    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:15:34.616717 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.616700    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:15:34.616787 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.616700    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:15:34.617739 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.617727    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:15:34.689632 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.689601    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkz98\" (UniqueName: \"kubernetes.io/projected/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-kube-api-access-vkz98\") pod \"maas-api-revoke-sub-57057b37-6rgvn\" (UID: \"dfbd370f-d4f3-440b-aa8d-e0d86efb9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"
Nov 02 12:15:34.689831 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.689812    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57057b37-6rgvn\" (UID: \"dfbd370f-d4f3-440b-aa8d-e0d86efb9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"
Nov 02 12:15:34.689902 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.689849    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57057b37-6rgvn\" (UID: \"dfbd370f-d4f3-440b-aa8d-e0d86efb9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"
Nov 02 12:15:34.690496 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.690474    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57057b37-6rgvn\" (UID: \"dfbd370f-d4f3-440b-aa8d-e0d86efb9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"
Nov 02 12:15:34.692055 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.692034    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57057b37-6rgvn\" (UID: \"dfbd370f-d4f3-440b-aa8d-e0d86efb9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"
Nov 02 12:15:34.751129 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.751098    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkz98\" (UniqueName: \"kubernetes.io/projected/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-kube-api-access-vkz98\") pod \"maas-api-revoke-sub-57057b37-6rgvn\" (UID: \"dfbd370f-d4f3-440b-aa8d-e0d86efb9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"
Nov 02 12:15:34.826164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:34.826086    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"
Nov 02 12:15:35.204308 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:35.204282    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"]
Nov 02 12:15:35.204426 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:15:35.204305    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfbd370f_d4f3_440b_aa8d_e0d86efb9475.slice/crio-97a686945f0fc5ae46e0e71912812b60107a811540038045545e00a941f973e7 WatchSource:0}: Error finding container 97a686945f0fc5ae46e0e71912812b60107a811540038045545e00a941f973e7: Status 404 returned error can't find the container with id 97a686945f0fc5ae46e0e71912812b60107a811540038045545e00a941f973e7
Nov 02 12:15:35.327837 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:35.327805    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"]
Nov 02 12:15:35.333172 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:35.333152    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"
Nov 02 12:15:35.389853 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:35.389826    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"]
Nov 02 12:15:35.400753 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:35.400718    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ccc71d1-lr4jt\" (UID: \"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"
Nov 02 12:15:35.400862 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:35.400763    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjdpv\" (UniqueName: \"kubernetes.io/projected/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-kube-api-access-tjdpv\") pod \"maas-api-revoke-sub-3ccc71d1-lr4jt\" (UID: \"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"
Nov 02 12:15:35.400862 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:35.400798    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ccc71d1-lr4jt\" (UID: \"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"
Nov 02 12:15:35.502357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:35.502322    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ccc71d1-lr4jt\" (UID: \"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"
Nov 02 12:15:35.502357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:35.502366    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjdpv\" (UniqueName: \"kubernetes.io/projected/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-kube-api-access-tjdpv\") pod \"maas-api-revoke-sub-3ccc71d1-lr4jt\" (UID: \"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"
Nov 02 12:15:35.502552 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:35.502540    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ccc71d1-lr4jt\" (UID: \"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"
Nov 02 12:15:35.503005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:35.502967    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ccc71d1-lr4jt\" (UID: \"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"
Nov 02 12:15:35.504603 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:35.504580    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ccc71d1-lr4jt\" (UID: \"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"
Nov 02 12:15:35.564434 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:35.564413    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjdpv\" (UniqueName: \"kubernetes.io/projected/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-kube-api-access-tjdpv\") pod \"maas-api-revoke-sub-3ccc71d1-lr4jt\" (UID: \"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"
Nov 02 12:15:35.646865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:35.646840    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"
Nov 02 12:15:36.021644 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:36.021612    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"]
Nov 02 12:15:36.021848 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:15:36.021827    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5fb801_5477_4f92_9ac7_8d65cf7df5d0.slice/crio-f29f3a4617be86efe461b814669bf8519a44aecba72edf8e845604e111ab49e5 WatchSource:0}: Error finding container f29f3a4617be86efe461b814669bf8519a44aecba72edf8e845604e111ab49e5: Status 404 returned error can't find the container with id f29f3a4617be86efe461b814669bf8519a44aecba72edf8e845604e111ab49e5
Nov 02 12:15:36.170111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:36.170076    4464 generic.go:356] "Generic (PLEG): container finished" podID="dfbd370f-d4f3-440b-aa8d-e0d86efb9475" containerID="11dac207c29ddeea5d0be147e68d4b85da562c97d41eb411642095f1bcd6a373" exitCode=0
Nov 02 12:15:36.170241 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:36.170167    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn" event={"ID":"dfbd370f-d4f3-440b-aa8d-e0d86efb9475","Type":"ContainerDied","Data":"11dac207c29ddeea5d0be147e68d4b85da562c97d41eb411642095f1bcd6a373"}
Nov 02 12:15:36.170241 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:36.170228    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn" event={"ID":"dfbd370f-d4f3-440b-aa8d-e0d86efb9475","Type":"ContainerStarted","Data":"97a686945f0fc5ae46e0e71912812b60107a811540038045545e00a941f973e7"}
Nov 02 12:15:36.171571 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:36.171549    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt" event={"ID":"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0","Type":"ContainerStarted","Data":"f29f3a4617be86efe461b814669bf8519a44aecba72edf8e845604e111ab49e5"}
Nov 02 12:15:37.179799 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:37.179725    4464 generic.go:356] "Generic (PLEG): container finished" podID="7d5fb801-5477-4f92-9ac7-8d65cf7df5d0" containerID="0876770036ee8d7b5d2bb6671b1348bdb75e47e98d0904500e7014404cf1eeb9" exitCode=0
Nov 02 12:15:37.180202 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:37.179808    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt" event={"ID":"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0","Type":"ContainerDied","Data":"0876770036ee8d7b5d2bb6671b1348bdb75e47e98d0904500e7014404cf1eeb9"}
Nov 02 12:15:37.341964 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:37.341940    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"
Nov 02 12:15:37.424382 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:37.424352    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-kube-api-access-vkz98\" (UniqueName: \"kubernetes.io/projected/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-kube-api-access-vkz98\") pod \"dfbd370f-d4f3-440b-aa8d-e0d86efb9475\" (UID: \"dfbd370f-d4f3-440b-aa8d-e0d86efb9475\") "
Nov 02 12:15:37.424525 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:37.424396    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-maas-api-openshift-ca\") pod \"dfbd370f-d4f3-440b-aa8d-e0d86efb9475\" (UID: \"dfbd370f-d4f3-440b-aa8d-e0d86efb9475\") "
Nov 02 12:15:37.424525 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:37.424438    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-maas-api-xks-ca\") pod \"dfbd370f-d4f3-440b-aa8d-e0d86efb9475\" (UID: \"dfbd370f-d4f3-440b-aa8d-e0d86efb9475\") "
Nov 02 12:15:37.424777 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:37.424748    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-maas-api-openshift-ca" pod "dfbd370f-d4f3-440b-aa8d-e0d86efb9475" (UID: "dfbd370f-d4f3-440b-aa8d-e0d86efb9475"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:15:37.424949 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:37.424932    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:37.426484 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:37.426457    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-kube-api-access-vkz98" pod "dfbd370f-d4f3-440b-aa8d-e0d86efb9475" (UID: "dfbd370f-d4f3-440b-aa8d-e0d86efb9475"). InnerVolumeSpecName "kube-api-access-vkz98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:15:37.426570 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:37.426515    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-maas-api-xks-ca" pod "dfbd370f-d4f3-440b-aa8d-e0d86efb9475" (UID: "dfbd370f-d4f3-440b-aa8d-e0d86efb9475"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:15:37.525765 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:37.525731    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkz98\" (UniqueName: \"kubernetes.io/projected/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-kube-api-access-vkz98\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:37.525765 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:37.525761    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfbd370f-d4f3-440b-aa8d-e0d86efb9475-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:38.187667 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:38.187639    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"
Nov 02 12:15:38.188178 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:38.187676    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn" event={"ID":"dfbd370f-d4f3-440b-aa8d-e0d86efb9475","Type":"ContainerDied","Data":"97a686945f0fc5ae46e0e71912812b60107a811540038045545e00a941f973e7"}
Nov 02 12:15:38.188178 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:38.187725    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97a686945f0fc5ae46e0e71912812b60107a811540038045545e00a941f973e7"
Nov 02 12:15:38.333895 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:38.333869    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"
Nov 02 12:15:38.432714 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:38.432689    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-kube-api-access-tjdpv\" (UniqueName: \"kubernetes.io/projected/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-kube-api-access-tjdpv\") pod \"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0\" (UID: \"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0\") "
Nov 02 12:15:38.432847 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:38.432793    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-maas-api-openshift-ca\") pod \"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0\" (UID: \"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0\") "
Nov 02 12:15:38.432847 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:38.432835    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-maas-api-xks-ca\") pod \"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0\" (UID: \"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0\") "
Nov 02 12:15:38.433118 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:38.433098    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-maas-api-openshift-ca" pod "7d5fb801-5477-4f92-9ac7-8d65cf7df5d0" (UID: "7d5fb801-5477-4f92-9ac7-8d65cf7df5d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:15:38.433386 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:38.433362    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:38.434785 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:38.434762    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-kube-api-access-tjdpv" pod "7d5fb801-5477-4f92-9ac7-8d65cf7df5d0" (UID: "7d5fb801-5477-4f92-9ac7-8d65cf7df5d0"). InnerVolumeSpecName "kube-api-access-tjdpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:15:38.434871 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:38.434837    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-maas-api-xks-ca" pod "7d5fb801-5477-4f92-9ac7-8d65cf7df5d0" (UID: "7d5fb801-5477-4f92-9ac7-8d65cf7df5d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:15:38.534686 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:38.534658    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:38.534686 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:38.534683    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjdpv\" (UniqueName: \"kubernetes.io/projected/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0-kube-api-access-tjdpv\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:39.196182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:39.196144    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"
Nov 02 12:15:39.196560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:39.196144    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt" event={"ID":"7d5fb801-5477-4f92-9ac7-8d65cf7df5d0","Type":"ContainerDied","Data":"f29f3a4617be86efe461b814669bf8519a44aecba72edf8e845604e111ab49e5"}
Nov 02 12:15:39.196560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:39.196296    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f29f3a4617be86efe461b814669bf8519a44aecba72edf8e845604e111ab49e5"
Nov 02 12:15:43.740997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.732822    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"]
Nov 02 12:15:43.755668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.753798    4464 cpu_manager.go:414] "containerMap: removing container" podUID="dfbd370f-d4f3-440b-aa8d-e0d86efb9475" containerName="revoke-keys"
Nov 02 12:15:43.755668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.753832    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfbd370f-d4f3-440b-aa8d-e0d86efb9475" containerName="revoke-keys"
Nov 02 12:15:43.755668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.753903    4464 cpu_manager.go:414] "containerMap: removing container" podUID="7d5fb801-5477-4f92-9ac7-8d65cf7df5d0" containerName="revoke-keys"
Nov 02 12:15:43.755668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.753910    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5fb801-5477-4f92-9ac7-8d65cf7df5d0" containerName="revoke-keys"
Nov 02 12:15:43.755668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.754051    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5fb801-5477-4f92-9ac7-8d65cf7df5d0" containerName="revoke-keys"
Nov 02 12:15:43.755668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.754066    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfbd370f-d4f3-440b-aa8d-e0d86efb9475" containerName="revoke-keys"
Nov 02 12:15:43.766388 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.765233    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"
Nov 02 12:15:43.788997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.784783    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"]
Nov 02 12:15:43.816852 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.816809    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn28g\" (UniqueName: \"kubernetes.io/projected/8363dde6-c7a7-422a-8d65-4999d48201d3-kube-api-access-tn28g\") pod \"maas-api-revoke-sub-c9924385-kngsf\" (UID: \"8363dde6-c7a7-422a-8d65-4999d48201d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"
Nov 02 12:15:43.817097 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.817019    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8363dde6-c7a7-422a-8d65-4999d48201d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9924385-kngsf\" (UID: \"8363dde6-c7a7-422a-8d65-4999d48201d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"
Nov 02 12:15:43.817097 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.817062    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8363dde6-c7a7-422a-8d65-4999d48201d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9924385-kngsf\" (UID: \"8363dde6-c7a7-422a-8d65-4999d48201d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"
Nov 02 12:15:43.829214 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.829189    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:15:43.829338 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.829224    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:15:43.829338 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.829324    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:15:43.870213 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.870191    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:15:43.918538 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.918510    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn28g\" (UniqueName: \"kubernetes.io/projected/8363dde6-c7a7-422a-8d65-4999d48201d3-kube-api-access-tn28g\") pod \"maas-api-revoke-sub-c9924385-kngsf\" (UID: \"8363dde6-c7a7-422a-8d65-4999d48201d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"
Nov 02 12:15:43.918683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.918597    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8363dde6-c7a7-422a-8d65-4999d48201d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9924385-kngsf\" (UID: \"8363dde6-c7a7-422a-8d65-4999d48201d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"
Nov 02 12:15:43.918683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.918622    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8363dde6-c7a7-422a-8d65-4999d48201d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9924385-kngsf\" (UID: \"8363dde6-c7a7-422a-8d65-4999d48201d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"
Nov 02 12:15:43.919272 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.919251    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8363dde6-c7a7-422a-8d65-4999d48201d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9924385-kngsf\" (UID: \"8363dde6-c7a7-422a-8d65-4999d48201d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"
Nov 02 12:15:43.920942 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.920919    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8363dde6-c7a7-422a-8d65-4999d48201d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9924385-kngsf\" (UID: \"8363dde6-c7a7-422a-8d65-4999d48201d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"
Nov 02 12:15:43.981095 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:43.981065    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn28g\" (UniqueName: \"kubernetes.io/projected/8363dde6-c7a7-422a-8d65-4999d48201d3-kube-api-access-tn28g\") pod \"maas-api-revoke-sub-c9924385-kngsf\" (UID: \"8363dde6-c7a7-422a-8d65-4999d48201d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"
Nov 02 12:15:44.109546 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:44.109181    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"
Nov 02 12:15:44.585631 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:15:44.585592    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8363dde6_c7a7_422a_8d65_4999d48201d3.slice/crio-927cc4247f980fc25833ce29f7f8a1ea9f4637ef3c128104c85d0e3d76b411bf WatchSource:0}: Error finding container 927cc4247f980fc25833ce29f7f8a1ea9f4637ef3c128104c85d0e3d76b411bf: Status 404 returned error can't find the container with id 927cc4247f980fc25833ce29f7f8a1ea9f4637ef3c128104c85d0e3d76b411bf
Nov 02 12:15:44.632387 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:44.632360    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"]
Nov 02 12:15:45.282998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:45.282833    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf" event={"ID":"8363dde6-c7a7-422a-8d65-4999d48201d3","Type":"ContainerStarted","Data":"0ce6d9cec5c37c73401231e0f7b871bc143237a84bdc30bbddef817051f010c6"}
Nov 02 12:15:45.282998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:45.282902    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf" event={"ID":"8363dde6-c7a7-422a-8d65-4999d48201d3","Type":"ContainerStarted","Data":"927cc4247f980fc25833ce29f7f8a1ea9f4637ef3c128104c85d0e3d76b411bf"}
Nov 02 12:15:46.328999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:46.328912    4464 generic.go:356] "Generic (PLEG): container finished" podID="8363dde6-c7a7-422a-8d65-4999d48201d3" containerID="0ce6d9cec5c37c73401231e0f7b871bc143237a84bdc30bbddef817051f010c6" exitCode=0
Nov 02 12:15:46.329515 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:46.329217    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf" event={"ID":"8363dde6-c7a7-422a-8d65-4999d48201d3","Type":"ContainerDied","Data":"0ce6d9cec5c37c73401231e0f7b871bc143237a84bdc30bbddef817051f010c6"}
Nov 02 12:15:46.386000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:46.368748    4464 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf" podUID="ff6c0679-4d25-499c-809d-4e22187c23ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:15:46.551000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:46.541063    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv" podUID="d829dab9-d274-485e-bef3-a29c5ddfff22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 02 12:15:47.740440 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:47.740408    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"
Nov 02 12:15:47.870002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:47.867462    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8363dde6-c7a7-422a-8d65-4999d48201d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8363dde6-c7a7-422a-8d65-4999d48201d3-maas-api-xks-ca\") pod \"8363dde6-c7a7-422a-8d65-4999d48201d3\" (UID: \"8363dde6-c7a7-422a-8d65-4999d48201d3\") "
Nov 02 12:15:47.870002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:47.867563    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8363dde6-c7a7-422a-8d65-4999d48201d3-kube-api-access-tn28g\" (UniqueName: \"kubernetes.io/projected/8363dde6-c7a7-422a-8d65-4999d48201d3-kube-api-access-tn28g\") pod \"8363dde6-c7a7-422a-8d65-4999d48201d3\" (UID: \"8363dde6-c7a7-422a-8d65-4999d48201d3\") "
Nov 02 12:15:47.870002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:47.867636    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8363dde6-c7a7-422a-8d65-4999d48201d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8363dde6-c7a7-422a-8d65-4999d48201d3-maas-api-openshift-ca\") pod \"8363dde6-c7a7-422a-8d65-4999d48201d3\" (UID: \"8363dde6-c7a7-422a-8d65-4999d48201d3\") "
Nov 02 12:15:47.870002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:47.868411    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8363dde6-c7a7-422a-8d65-4999d48201d3-maas-api-openshift-ca" pod "8363dde6-c7a7-422a-8d65-4999d48201d3" (UID: "8363dde6-c7a7-422a-8d65-4999d48201d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:15:47.894138 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:47.894105    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8363dde6-c7a7-422a-8d65-4999d48201d3-maas-api-xks-ca" pod "8363dde6-c7a7-422a-8d65-4999d48201d3" (UID: "8363dde6-c7a7-422a-8d65-4999d48201d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:15:47.895171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:47.895141    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8363dde6-c7a7-422a-8d65-4999d48201d3-kube-api-access-tn28g" pod "8363dde6-c7a7-422a-8d65-4999d48201d3" (UID: "8363dde6-c7a7-422a-8d65-4999d48201d3"). InnerVolumeSpecName "kube-api-access-tn28g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:15:47.983045 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:47.983006    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8363dde6-c7a7-422a-8d65-4999d48201d3-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:47.983045 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:47.983047    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8363dde6-c7a7-422a-8d65-4999d48201d3-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:47.983280 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:47.983064    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn28g\" (UniqueName: \"kubernetes.io/projected/8363dde6-c7a7-422a-8d65-4999d48201d3-kube-api-access-tn28g\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:48.379846 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:48.379721    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf" event={"ID":"8363dde6-c7a7-422a-8d65-4999d48201d3","Type":"ContainerDied","Data":"927cc4247f980fc25833ce29f7f8a1ea9f4637ef3c128104c85d0e3d76b411bf"}
Nov 02 12:15:48.380339 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:48.380311    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="927cc4247f980fc25833ce29f7f8a1ea9f4637ef3c128104c85d0e3d76b411bf"
Nov 02 12:15:48.380497 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:48.380122    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"
Nov 02 12:15:51.143940 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.143909    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"]
Nov 02 12:15:51.144581 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.144558    4464 cpu_manager.go:414] "containerMap: removing container" podUID="8363dde6-c7a7-422a-8d65-4999d48201d3" containerName="revoke-keys"
Nov 02 12:15:51.144673 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.144581    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8363dde6-c7a7-422a-8d65-4999d48201d3" containerName="revoke-keys"
Nov 02 12:15:51.144756 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.144743    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="8363dde6-c7a7-422a-8d65-4999d48201d3" containerName="revoke-keys"
Nov 02 12:15:51.149203 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.149186    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"
Nov 02 12:15:51.205984 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.202659    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:15:51.206664 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.206639    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"]
Nov 02 12:15:51.220601 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.220573    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f9bd26-5764-4ce1-99a3-f760b195a2f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1649951e-zn8vp\" (UID: \"61f9bd26-5764-4ce1-99a3-f760b195a2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"
Nov 02 12:15:51.220747 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.220725    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f9bd26-5764-4ce1-99a3-f760b195a2f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1649951e-zn8vp\" (UID: \"61f9bd26-5764-4ce1-99a3-f760b195a2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"
Nov 02 12:15:51.220817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.220752    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8mk6\" (UniqueName: \"kubernetes.io/projected/61f9bd26-5764-4ce1-99a3-f760b195a2f5-kube-api-access-s8mk6\") pod \"maas-api-revoke-sub-1649951e-zn8vp\" (UID: \"61f9bd26-5764-4ce1-99a3-f760b195a2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"
Nov 02 12:15:51.250396 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.250367    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:15:51.250396 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.250391    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:15:51.252523 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.252502    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:15:51.322029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.321970    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f9bd26-5764-4ce1-99a3-f760b195a2f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1649951e-zn8vp\" (UID: \"61f9bd26-5764-4ce1-99a3-f760b195a2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"
Nov 02 12:15:51.322150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.322066    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8mk6\" (UniqueName: \"kubernetes.io/projected/61f9bd26-5764-4ce1-99a3-f760b195a2f5-kube-api-access-s8mk6\") pod \"maas-api-revoke-sub-1649951e-zn8vp\" (UID: \"61f9bd26-5764-4ce1-99a3-f760b195a2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"
Nov 02 12:15:51.322241 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.322226    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f9bd26-5764-4ce1-99a3-f760b195a2f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1649951e-zn8vp\" (UID: \"61f9bd26-5764-4ce1-99a3-f760b195a2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"
Nov 02 12:15:51.322856 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.322835    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f9bd26-5764-4ce1-99a3-f760b195a2f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1649951e-zn8vp\" (UID: \"61f9bd26-5764-4ce1-99a3-f760b195a2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"
Nov 02 12:15:51.324196 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.324172    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f9bd26-5764-4ce1-99a3-f760b195a2f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1649951e-zn8vp\" (UID: \"61f9bd26-5764-4ce1-99a3-f760b195a2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"
Nov 02 12:15:51.384075 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.384019    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8mk6\" (UniqueName: \"kubernetes.io/projected/61f9bd26-5764-4ce1-99a3-f760b195a2f5-kube-api-access-s8mk6\") pod \"maas-api-revoke-sub-1649951e-zn8vp\" (UID: \"61f9bd26-5764-4ce1-99a3-f760b195a2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"
Nov 02 12:15:51.466934 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.466906    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"
Nov 02 12:15:51.848394 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:51.848362    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"]
Nov 02 12:15:51.849090 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:15:51.849058    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f9bd26_5764_4ce1_99a3_f760b195a2f5.slice/crio-0041da8f98fc7203dc66491fba115400eade29e46cb2fb3054abfff6dc8d0eb2 WatchSource:0}: Error finding container 0041da8f98fc7203dc66491fba115400eade29e46cb2fb3054abfff6dc8d0eb2: Status 404 returned error can't find the container with id 0041da8f98fc7203dc66491fba115400eade29e46cb2fb3054abfff6dc8d0eb2
Nov 02 12:15:52.418626 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:52.418576    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp" event={"ID":"61f9bd26-5764-4ce1-99a3-f760b195a2f5","Type":"ContainerStarted","Data":"80ff7eb703ee2544a07fc7e7ac86ef185ebaf035b68a651713e320e94cfc059e"}
Nov 02 12:15:52.419138 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:52.418699    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp" event={"ID":"61f9bd26-5764-4ce1-99a3-f760b195a2f5","Type":"ContainerStarted","Data":"0041da8f98fc7203dc66491fba115400eade29e46cb2fb3054abfff6dc8d0eb2"}
Nov 02 12:15:52.474058 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:52.474024    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"]
Nov 02 12:15:52.478621 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:52.478600    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"
Nov 02 12:15:52.537746 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:52.537626    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca53aea4-d372-4da2-b157-f74ef01dcfb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a9451f-kl2hf\" (UID: \"ca53aea4-d372-4da2-b157-f74ef01dcfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"
Nov 02 12:15:52.538000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:52.537968    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca53aea4-d372-4da2-b157-f74ef01dcfb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a9451f-kl2hf\" (UID: \"ca53aea4-d372-4da2-b157-f74ef01dcfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"
Nov 02 12:15:52.538111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:52.538098    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trhgb\" (UniqueName: \"kubernetes.io/projected/ca53aea4-d372-4da2-b157-f74ef01dcfb7-kube-api-access-trhgb\") pod \"maas-api-revoke-sub-a5a9451f-kl2hf\" (UID: \"ca53aea4-d372-4da2-b157-f74ef01dcfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"
Nov 02 12:15:52.544248 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:52.544175    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"]
Nov 02 12:15:52.544684 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:52.544643    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp" podStartSLOduration=1.544630642 podStartE2EDuration="1.544630642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:15:52.536766591 +0000 UTC m=+1181.100475013" watchObservedRunningTime="2026-11-02 12:15:52.544630642 +0000 UTC m=+1181.108339062"
Nov 02 12:15:52.639769 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:52.639727    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca53aea4-d372-4da2-b157-f74ef01dcfb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a9451f-kl2hf\" (UID: \"ca53aea4-d372-4da2-b157-f74ef01dcfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"
Nov 02 12:15:52.640004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:52.639792    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca53aea4-d372-4da2-b157-f74ef01dcfb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a9451f-kl2hf\" (UID: \"ca53aea4-d372-4da2-b157-f74ef01dcfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"
Nov 02 12:15:52.640004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:52.639822    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trhgb\" (UniqueName: \"kubernetes.io/projected/ca53aea4-d372-4da2-b157-f74ef01dcfb7-kube-api-access-trhgb\") pod \"maas-api-revoke-sub-a5a9451f-kl2hf\" (UID: \"ca53aea4-d372-4da2-b157-f74ef01dcfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"
Nov 02 12:15:52.640564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:52.640542    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca53aea4-d372-4da2-b157-f74ef01dcfb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a9451f-kl2hf\" (UID: \"ca53aea4-d372-4da2-b157-f74ef01dcfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"
Nov 02 12:15:52.642394 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:52.642368    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca53aea4-d372-4da2-b157-f74ef01dcfb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a9451f-kl2hf\" (UID: \"ca53aea4-d372-4da2-b157-f74ef01dcfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"
Nov 02 12:15:52.702584 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:52.702548    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trhgb\" (UniqueName: \"kubernetes.io/projected/ca53aea4-d372-4da2-b157-f74ef01dcfb7-kube-api-access-trhgb\") pod \"maas-api-revoke-sub-a5a9451f-kl2hf\" (UID: \"ca53aea4-d372-4da2-b157-f74ef01dcfb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"
Nov 02 12:15:52.794093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:52.794016    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"
Nov 02 12:15:53.174711 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:15:53.174680    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca53aea4_d372_4da2_b157_f74ef01dcfb7.slice/crio-4886ea714bc5728b4080bd437bbb1333ebc2dda91e4bd74c0dc77798690f39ab WatchSource:0}: Error finding container 4886ea714bc5728b4080bd437bbb1333ebc2dda91e4bd74c0dc77798690f39ab: Status 404 returned error can't find the container with id 4886ea714bc5728b4080bd437bbb1333ebc2dda91e4bd74c0dc77798690f39ab
Nov 02 12:15:53.174835 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:53.174786    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"]
Nov 02 12:15:53.427182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:53.427104    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf" event={"ID":"ca53aea4-d372-4da2-b157-f74ef01dcfb7","Type":"ContainerStarted","Data":"4886ea714bc5728b4080bd437bbb1333ebc2dda91e4bd74c0dc77798690f39ab"}
Nov 02 12:15:53.428774 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:53.428744    4464 generic.go:356] "Generic (PLEG): container finished" podID="61f9bd26-5764-4ce1-99a3-f760b195a2f5" containerID="80ff7eb703ee2544a07fc7e7ac86ef185ebaf035b68a651713e320e94cfc059e" exitCode=0
Nov 02 12:15:53.428890 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:53.428785    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp" event={"ID":"61f9bd26-5764-4ce1-99a3-f760b195a2f5","Type":"ContainerDied","Data":"80ff7eb703ee2544a07fc7e7ac86ef185ebaf035b68a651713e320e94cfc059e"}
Nov 02 12:15:54.437480 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.437401    4464 generic.go:356] "Generic (PLEG): container finished" podID="ca53aea4-d372-4da2-b157-f74ef01dcfb7" containerID="051b60ce4f07c18d05bec284c9f7aa5f2003401b39c27092849820a5d4d031fe" exitCode=0
Nov 02 12:15:54.437480 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.437446    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf" event={"ID":"ca53aea4-d372-4da2-b157-f74ef01dcfb7","Type":"ContainerDied","Data":"051b60ce4f07c18d05bec284c9f7aa5f2003401b39c27092849820a5d4d031fe"}
Nov 02 12:15:54.491827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.491797    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"]
Nov 02 12:15:54.495539 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.495333    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"
Nov 02 12:15:54.563462 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.563416    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"]
Nov 02 12:15:54.563777 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.563752    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f495d2bb-beda-4dba-b98d-584df140c13b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-798c44bf-76g4d\" (UID: \"f495d2bb-beda-4dba-b98d-584df140c13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"
Nov 02 12:15:54.563836 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.563814    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6q2l\" (UniqueName: \"kubernetes.io/projected/f495d2bb-beda-4dba-b98d-584df140c13b-kube-api-access-m6q2l\") pod \"maas-api-revoke-sub-798c44bf-76g4d\" (UID: \"f495d2bb-beda-4dba-b98d-584df140c13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"
Nov 02 12:15:54.564076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.564058    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f495d2bb-beda-4dba-b98d-584df140c13b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-798c44bf-76g4d\" (UID: \"f495d2bb-beda-4dba-b98d-584df140c13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"
Nov 02 12:15:54.647249 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.647227    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"
Nov 02 12:15:54.664970 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.664946    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f9bd26-5764-4ce1-99a3-f760b195a2f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f9bd26-5764-4ce1-99a3-f760b195a2f5-maas-api-openshift-ca\") pod \"61f9bd26-5764-4ce1-99a3-f760b195a2f5\" (UID: \"61f9bd26-5764-4ce1-99a3-f760b195a2f5\") "
Nov 02 12:15:54.665101 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.665033    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f9bd26-5764-4ce1-99a3-f760b195a2f5-kube-api-access-s8mk6\" (UniqueName: \"kubernetes.io/projected/61f9bd26-5764-4ce1-99a3-f760b195a2f5-kube-api-access-s8mk6\") pod \"61f9bd26-5764-4ce1-99a3-f760b195a2f5\" (UID: \"61f9bd26-5764-4ce1-99a3-f760b195a2f5\") "
Nov 02 12:15:54.665143 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.665099    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61f9bd26-5764-4ce1-99a3-f760b195a2f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f9bd26-5764-4ce1-99a3-f760b195a2f5-maas-api-xks-ca\") pod \"61f9bd26-5764-4ce1-99a3-f760b195a2f5\" (UID: \"61f9bd26-5764-4ce1-99a3-f760b195a2f5\") "
Nov 02 12:15:54.665240 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.665224    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f495d2bb-beda-4dba-b98d-584df140c13b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-798c44bf-76g4d\" (UID: \"f495d2bb-beda-4dba-b98d-584df140c13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"
Nov 02 12:15:54.665304 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.665264    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f9bd26-5764-4ce1-99a3-f760b195a2f5-maas-api-openshift-ca" pod "61f9bd26-5764-4ce1-99a3-f760b195a2f5" (UID: "61f9bd26-5764-4ce1-99a3-f760b195a2f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:15:54.665398 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.665382    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f495d2bb-beda-4dba-b98d-584df140c13b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-798c44bf-76g4d\" (UID: \"f495d2bb-beda-4dba-b98d-584df140c13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"
Nov 02 12:15:54.665473 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.665428    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6q2l\" (UniqueName: \"kubernetes.io/projected/f495d2bb-beda-4dba-b98d-584df140c13b-kube-api-access-m6q2l\") pod \"maas-api-revoke-sub-798c44bf-76g4d\" (UID: \"f495d2bb-beda-4dba-b98d-584df140c13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"
Nov 02 12:15:54.665679 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.665659    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61f9bd26-5764-4ce1-99a3-f760b195a2f5-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:54.665986 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.665957    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f495d2bb-beda-4dba-b98d-584df140c13b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-798c44bf-76g4d\" (UID: \"f495d2bb-beda-4dba-b98d-584df140c13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"
Nov 02 12:15:54.667137 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.667114    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f9bd26-5764-4ce1-99a3-f760b195a2f5-maas-api-xks-ca" pod "61f9bd26-5764-4ce1-99a3-f760b195a2f5" (UID: "61f9bd26-5764-4ce1-99a3-f760b195a2f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:15:54.667237 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.667216    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f9bd26-5764-4ce1-99a3-f760b195a2f5-kube-api-access-s8mk6" pod "61f9bd26-5764-4ce1-99a3-f760b195a2f5" (UID: "61f9bd26-5764-4ce1-99a3-f760b195a2f5"). InnerVolumeSpecName "kube-api-access-s8mk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:15:54.667590 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.667576    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f495d2bb-beda-4dba-b98d-584df140c13b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-798c44bf-76g4d\" (UID: \"f495d2bb-beda-4dba-b98d-584df140c13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"
Nov 02 12:15:54.727180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.727152    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6q2l\" (UniqueName: \"kubernetes.io/projected/f495d2bb-beda-4dba-b98d-584df140c13b-kube-api-access-m6q2l\") pod \"maas-api-revoke-sub-798c44bf-76g4d\" (UID: \"f495d2bb-beda-4dba-b98d-584df140c13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"
Nov 02 12:15:54.766467 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.766430    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8mk6\" (UniqueName: \"kubernetes.io/projected/61f9bd26-5764-4ce1-99a3-f760b195a2f5-kube-api-access-s8mk6\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:54.766467 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.766467    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61f9bd26-5764-4ce1-99a3-f760b195a2f5-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:54.824153 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:54.824123    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"
Nov 02 12:15:55.204237 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:15:55.204197    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf495d2bb_beda_4dba_b98d_584df140c13b.slice/crio-2718e5467626f1a80a9580dca3dbe4c5d43138cb1801bcf5e17acd907fd90f0c WatchSource:0}: Error finding container 2718e5467626f1a80a9580dca3dbe4c5d43138cb1801bcf5e17acd907fd90f0c: Status 404 returned error can't find the container with id 2718e5467626f1a80a9580dca3dbe4c5d43138cb1801bcf5e17acd907fd90f0c
Nov 02 12:15:55.204800 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.204727    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"]
Nov 02 12:15:55.445936 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.445910    4464 generic.go:356] "Generic (PLEG): container finished" podID="90e7a43f-8da1-4a73-ab5e-f5edd3573d80" containerID="1ff1b5d3ae12317a255583ef40209d3a74e3519e9a2e859154157afb75f9c2b8" exitCode=0
Nov 02 12:15:55.446363 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.445998    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2" event={"ID":"90e7a43f-8da1-4a73-ab5e-f5edd3573d80","Type":"ContainerDied","Data":"1ff1b5d3ae12317a255583ef40209d3a74e3519e9a2e859154157afb75f9c2b8"}
Nov 02 12:15:55.446424 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.446375    4464 scope.go:122] "RemoveContainer" containerID="1ff1b5d3ae12317a255583ef40209d3a74e3519e9a2e859154157afb75f9c2b8"
Nov 02 12:15:55.447472 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.447454    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"
Nov 02 12:15:55.447572 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.447460    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp" event={"ID":"61f9bd26-5764-4ce1-99a3-f760b195a2f5","Type":"ContainerDied","Data":"0041da8f98fc7203dc66491fba115400eade29e46cb2fb3054abfff6dc8d0eb2"}
Nov 02 12:15:55.447572 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.447508    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0041da8f98fc7203dc66491fba115400eade29e46cb2fb3054abfff6dc8d0eb2"
Nov 02 12:15:55.447719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.447698    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:15:55.447792 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.447775    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:15:55.447842 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.447827    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:15:55.448901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.448879    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d" event={"ID":"f495d2bb-beda-4dba-b98d-584df140c13b","Type":"ContainerStarted","Data":"2718e5467626f1a80a9580dca3dbe4c5d43138cb1801bcf5e17acd907fd90f0c"}
Nov 02 12:15:55.631843 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.631813    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"
Nov 02 12:15:55.679739 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.679705    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca53aea4-d372-4da2-b157-f74ef01dcfb7-kube-api-access-trhgb\" (UniqueName: \"kubernetes.io/projected/ca53aea4-d372-4da2-b157-f74ef01dcfb7-kube-api-access-trhgb\") pod \"ca53aea4-d372-4da2-b157-f74ef01dcfb7\" (UID: \"ca53aea4-d372-4da2-b157-f74ef01dcfb7\") "
Nov 02 12:15:55.680805 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.680781    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca53aea4-d372-4da2-b157-f74ef01dcfb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca53aea4-d372-4da2-b157-f74ef01dcfb7-maas-api-openshift-ca\") pod \"ca53aea4-d372-4da2-b157-f74ef01dcfb7\" (UID: \"ca53aea4-d372-4da2-b157-f74ef01dcfb7\") "
Nov 02 12:15:55.686998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.685753    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca53aea4-d372-4da2-b157-f74ef01dcfb7-maas-api-openshift-ca" pod "ca53aea4-d372-4da2-b157-f74ef01dcfb7" (UID: "ca53aea4-d372-4da2-b157-f74ef01dcfb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:15:55.686998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.685925    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca53aea4-d372-4da2-b157-f74ef01dcfb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca53aea4-d372-4da2-b157-f74ef01dcfb7-maas-api-xks-ca\") pod \"ca53aea4-d372-4da2-b157-f74ef01dcfb7\" (UID: \"ca53aea4-d372-4da2-b157-f74ef01dcfb7\") "
Nov 02 12:15:55.686998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.686670    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca53aea4-d372-4da2-b157-f74ef01dcfb7-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:55.689146 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.689119    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca53aea4-d372-4da2-b157-f74ef01dcfb7-kube-api-access-trhgb" pod "ca53aea4-d372-4da2-b157-f74ef01dcfb7" (UID: "ca53aea4-d372-4da2-b157-f74ef01dcfb7"). InnerVolumeSpecName "kube-api-access-trhgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:15:55.689267 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.689244    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca53aea4-d372-4da2-b157-f74ef01dcfb7-maas-api-xks-ca" pod "ca53aea4-d372-4da2-b157-f74ef01dcfb7" (UID: "ca53aea4-d372-4da2-b157-f74ef01dcfb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:15:55.787798 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.787765    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trhgb\" (UniqueName: \"kubernetes.io/projected/ca53aea4-d372-4da2-b157-f74ef01dcfb7-kube-api-access-trhgb\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:55.787798 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:55.787795    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca53aea4-d372-4da2-b157-f74ef01dcfb7-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:56.457802 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:56.457759    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2" event={"ID":"90e7a43f-8da1-4a73-ab5e-f5edd3573d80","Type":"ContainerStarted","Data":"b2289d6fc6cbaf7a0163f006462f31cabec0e3ee92e56a3e9e5af1fdcb85ba2e"}
Nov 02 12:15:56.481000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:56.479679    4464 generic.go:356] "Generic (PLEG): container finished" podID="f495d2bb-beda-4dba-b98d-584df140c13b" containerID="7c353570839f47f9063df7ed62593f02e20d5880870b84afddc8e76f6fcb4aa8" exitCode=0
Nov 02 12:15:56.481000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:56.480003    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d" event={"ID":"f495d2bb-beda-4dba-b98d-584df140c13b","Type":"ContainerDied","Data":"7c353570839f47f9063df7ed62593f02e20d5880870b84afddc8e76f6fcb4aa8"}
Nov 02 12:15:56.483996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:56.482058    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf" event={"ID":"ca53aea4-d372-4da2-b157-f74ef01dcfb7","Type":"ContainerDied","Data":"4886ea714bc5728b4080bd437bbb1333ebc2dda91e4bd74c0dc77798690f39ab"}
Nov 02 12:15:56.483996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:56.482105    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4886ea714bc5728b4080bd437bbb1333ebc2dda91e4bd74c0dc77798690f39ab"
Nov 02 12:15:56.483996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:56.482167    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"
Nov 02 12:15:56.726386 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:56.726359    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:15:56.727000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:56.726966    4464 patch_prober.go:29] interesting pod/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 12:15:56.727106 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:56.727028    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2" podUID="90e7a43f-8da1-4a73-ab5e-f5edd3573d80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 12:15:56.954323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:56.954295    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:15:57.745004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:57.741630    4464 patch_prober.go:29] interesting pod/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 12:15:57.745004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:57.741696    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2" podUID="90e7a43f-8da1-4a73-ab5e-f5edd3573d80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 12:15:58.335305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:58.335223    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"
Nov 02 12:15:58.440472 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:58.440438    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f495d2bb-beda-4dba-b98d-584df140c13b-kube-api-access-m6q2l\" (UniqueName: \"kubernetes.io/projected/f495d2bb-beda-4dba-b98d-584df140c13b-kube-api-access-m6q2l\") pod \"f495d2bb-beda-4dba-b98d-584df140c13b\" (UID: \"f495d2bb-beda-4dba-b98d-584df140c13b\") "
Nov 02 12:15:58.441169 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:58.441148    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f495d2bb-beda-4dba-b98d-584df140c13b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f495d2bb-beda-4dba-b98d-584df140c13b-maas-api-openshift-ca\") pod \"f495d2bb-beda-4dba-b98d-584df140c13b\" (UID: \"f495d2bb-beda-4dba-b98d-584df140c13b\") "
Nov 02 12:15:58.451267 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:58.451165    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f495d2bb-beda-4dba-b98d-584df140c13b-maas-api-openshift-ca" pod "f495d2bb-beda-4dba-b98d-584df140c13b" (UID: "f495d2bb-beda-4dba-b98d-584df140c13b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:15:58.451430 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:58.451260    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f495d2bb-beda-4dba-b98d-584df140c13b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f495d2bb-beda-4dba-b98d-584df140c13b-maas-api-xks-ca\") pod \"f495d2bb-beda-4dba-b98d-584df140c13b\" (UID: \"f495d2bb-beda-4dba-b98d-584df140c13b\") "
Nov 02 12:15:58.459542 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:58.459518    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f495d2bb-beda-4dba-b98d-584df140c13b-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:58.541034 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:58.538856    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f495d2bb-beda-4dba-b98d-584df140c13b-kube-api-access-m6q2l" pod "f495d2bb-beda-4dba-b98d-584df140c13b" (UID: "f495d2bb-beda-4dba-b98d-584df140c13b"). InnerVolumeSpecName "kube-api-access-m6q2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:15:58.569203 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:58.569135    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f495d2bb-beda-4dba-b98d-584df140c13b-maas-api-xks-ca" pod "f495d2bb-beda-4dba-b98d-584df140c13b" (UID: "f495d2bb-beda-4dba-b98d-584df140c13b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:15:58.601083 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:58.579916    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d" event={"ID":"f495d2bb-beda-4dba-b98d-584df140c13b","Type":"ContainerDied","Data":"2718e5467626f1a80a9580dca3dbe4c5d43138cb1801bcf5e17acd907fd90f0c"}
Nov 02 12:15:58.601083 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:58.579991    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2718e5467626f1a80a9580dca3dbe4c5d43138cb1801bcf5e17acd907fd90f0c"
Nov 02 12:15:58.601083 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:58.580057    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"
Nov 02 12:15:58.661996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:58.602875    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6q2l\" (UniqueName: \"kubernetes.io/projected/f495d2bb-beda-4dba-b98d-584df140c13b-kube-api-access-m6q2l\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:58.661996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:58.602914    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f495d2bb-beda-4dba-b98d-584df140c13b-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:15:58.793305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:58.793267    4464 patch_prober.go:29] interesting pod/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 12:15:58.793908 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:58.793875    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2" podUID="90e7a43f-8da1-4a73-ab5e-f5edd3573d80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 12:15:58.857468 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:15:58.857372    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:15:59.540999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:59.538633    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv" podUID="d829dab9-d274-485e-bef3-a29c5ddfff22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:15:59.606005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:59.605929    4464 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv" podUID="d829dab9-d274-485e-bef3-a29c5ddfff22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:15:59.731404 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:59.731369    4464 patch_prober.go:29] interesting pod/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 12:15:59.731668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:59.731640    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2" podUID="90e7a43f-8da1-4a73-ab5e-f5edd3573d80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 12:15:59.875240 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:59.875144    4464 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72" podUID="df859e10-37e4-4d2e-a1d5-931adacbaaff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:15:59.876119 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:15:59.876084    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf" podUID="ff6c0679-4d25-499c-809d-4e22187c23ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:16:00.030122 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:00.030062    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72" podUID="df859e10-37e4-4d2e-a1d5-931adacbaaff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:16:00.545001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:00.541177    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv" podUID="d829dab9-d274-485e-bef3-a29c5ddfff22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 02 12:16:00.545001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:00.541926    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:16:00.741815 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:00.741778    4464 patch_prober.go:29] interesting pod/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 12:16:00.744963 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:00.744927    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2" podUID="90e7a43f-8da1-4a73-ab5e-f5edd3573d80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 12:16:00.850050 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:00.835883    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:16:01.725851 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:01.725814    4464 patch_prober.go:29] interesting pod/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 12:16:01.726266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:01.725879    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2" podUID="90e7a43f-8da1-4a73-ab5e-f5edd3573d80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 12:16:01.928117 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:01.928081    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"]
Nov 02 12:16:01.928668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:01.928655    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f495d2bb-beda-4dba-b98d-584df140c13b" containerName="revoke-keys"
Nov 02 12:16:01.928719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:01.928670    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f495d2bb-beda-4dba-b98d-584df140c13b" containerName="revoke-keys"
Nov 02 12:16:01.928719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:01.928695    4464 cpu_manager.go:414] "containerMap: removing container" podUID="61f9bd26-5764-4ce1-99a3-f760b195a2f5" containerName="revoke-keys"
Nov 02 12:16:01.928719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:01.928700    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f9bd26-5764-4ce1-99a3-f760b195a2f5" containerName="revoke-keys"
Nov 02 12:16:01.928719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:01.928711    4464 cpu_manager.go:414] "containerMap: removing container" podUID="ca53aea4-d372-4da2-b157-f74ef01dcfb7" containerName="revoke-keys"
Nov 02 12:16:01.928719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:01.928716    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca53aea4-d372-4da2-b157-f74ef01dcfb7" containerName="revoke-keys"
Nov 02 12:16:01.928872 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:01.928800    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f495d2bb-beda-4dba-b98d-584df140c13b" containerName="revoke-keys"
Nov 02 12:16:01.928872 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:01.928813    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f9bd26-5764-4ce1-99a3-f760b195a2f5" containerName="revoke-keys"
Nov 02 12:16:01.928872 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:01.928820    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca53aea4-d372-4da2-b157-f74ef01dcfb7" containerName="revoke-keys"
Nov 02 12:16:01.936927 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:01.936894    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"
Nov 02 12:16:01.988051 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:01.987949    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ae79f87-f000-4903-a930-aa9a5f12c324-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d10425d1-jql9m\" (UID: \"8ae79f87-f000-4903-a930-aa9a5f12c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"
Nov 02 12:16:01.991572 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:01.991532    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ae79f87-f000-4903-a930-aa9a5f12c324-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d10425d1-jql9m\" (UID: \"8ae79f87-f000-4903-a930-aa9a5f12c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"
Nov 02 12:16:01.991689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:01.991670    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx4jz\" (UniqueName: \"kubernetes.io/projected/8ae79f87-f000-4903-a930-aa9a5f12c324-kube-api-access-vx4jz\") pod \"maas-api-revoke-sub-d10425d1-jql9m\" (UID: \"8ae79f87-f000-4903-a930-aa9a5f12c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"
Nov 02 12:16:01.994102 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:01.994078    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"]
Nov 02 12:16:02.036956 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:02.036929    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:16:02.036956 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:02.036943    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:16:02.036956 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:02.036941    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:16:02.040682 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:02.040663    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:16:02.092802 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:02.092774    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ae79f87-f000-4903-a930-aa9a5f12c324-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d10425d1-jql9m\" (UID: \"8ae79f87-f000-4903-a930-aa9a5f12c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"
Nov 02 12:16:02.093052 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:02.093037    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx4jz\" (UniqueName: \"kubernetes.io/projected/8ae79f87-f000-4903-a930-aa9a5f12c324-kube-api-access-vx4jz\") pod \"maas-api-revoke-sub-d10425d1-jql9m\" (UID: \"8ae79f87-f000-4903-a930-aa9a5f12c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"
Nov 02 12:16:02.093237 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:02.093225    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ae79f87-f000-4903-a930-aa9a5f12c324-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d10425d1-jql9m\" (UID: \"8ae79f87-f000-4903-a930-aa9a5f12c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"
Nov 02 12:16:02.093524 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:02.093504    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ae79f87-f000-4903-a930-aa9a5f12c324-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d10425d1-jql9m\" (UID: \"8ae79f87-f000-4903-a930-aa9a5f12c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"
Nov 02 12:16:02.095421 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:02.095400    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ae79f87-f000-4903-a930-aa9a5f12c324-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d10425d1-jql9m\" (UID: \"8ae79f87-f000-4903-a930-aa9a5f12c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"
Nov 02 12:16:02.154582 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:02.154557    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx4jz\" (UniqueName: \"kubernetes.io/projected/8ae79f87-f000-4903-a930-aa9a5f12c324-kube-api-access-vx4jz\") pod \"maas-api-revoke-sub-d10425d1-jql9m\" (UID: \"8ae79f87-f000-4903-a930-aa9a5f12c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"
Nov 02 12:16:02.256480 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:02.256406    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"
Nov 02 12:16:02.664006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:02.663949    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"]
Nov 02 12:16:02.666851 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:16:02.666817    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae79f87_f000_4903_a930_aa9a5f12c324.slice/crio-b5b37a50920a0503368e60d3144ba96128ef6cedd79d4824765c5ba68f594d1f WatchSource:0}: Error finding container b5b37a50920a0503368e60d3144ba96128ef6cedd79d4824765c5ba68f594d1f: Status 404 returned error can't find the container with id b5b37a50920a0503368e60d3144ba96128ef6cedd79d4824765c5ba68f594d1f
Nov 02 12:16:02.700579 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:02.700552    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m" event={"ID":"8ae79f87-f000-4903-a930-aa9a5f12c324","Type":"ContainerStarted","Data":"b5b37a50920a0503368e60d3144ba96128ef6cedd79d4824765c5ba68f594d1f"}
Nov 02 12:16:02.770582 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:02.770555    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:16:02.772725 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:02.772702    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:16:02.780999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:02.774114    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 12:16:03.708941 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:03.708908    4464 generic.go:356] "Generic (PLEG): container finished" podID="8ae79f87-f000-4903-a930-aa9a5f12c324" containerID="22fb6b054c6101444093e3017b5d217e01111fc0e9b801de6b3991f564b07421" exitCode=0
Nov 02 12:16:03.709118 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:03.709007    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m" event={"ID":"8ae79f87-f000-4903-a930-aa9a5f12c324","Type":"ContainerDied","Data":"22fb6b054c6101444093e3017b5d217e01111fc0e9b801de6b3991f564b07421"}
Nov 02 12:16:04.868040 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:04.868015    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"
Nov 02 12:16:04.926406 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:04.926378    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ae79f87-f000-4903-a930-aa9a5f12c324-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ae79f87-f000-4903-a930-aa9a5f12c324-maas-api-xks-ca\") pod \"8ae79f87-f000-4903-a930-aa9a5f12c324\" (UID: \"8ae79f87-f000-4903-a930-aa9a5f12c324\") "
Nov 02 12:16:04.926545 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:04.926433    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ae79f87-f000-4903-a930-aa9a5f12c324-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ae79f87-f000-4903-a930-aa9a5f12c324-maas-api-openshift-ca\") pod \"8ae79f87-f000-4903-a930-aa9a5f12c324\" (UID: \"8ae79f87-f000-4903-a930-aa9a5f12c324\") "
Nov 02 12:16:04.926625 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:04.926542    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae79f87-f000-4903-a930-aa9a5f12c324-kube-api-access-vx4jz\" (UniqueName: \"kubernetes.io/projected/8ae79f87-f000-4903-a930-aa9a5f12c324-kube-api-access-vx4jz\") pod \"8ae79f87-f000-4903-a930-aa9a5f12c324\" (UID: \"8ae79f87-f000-4903-a930-aa9a5f12c324\") "
Nov 02 12:16:04.926849 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:04.926822    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae79f87-f000-4903-a930-aa9a5f12c324-maas-api-openshift-ca" pod "8ae79f87-f000-4903-a930-aa9a5f12c324" (UID: "8ae79f87-f000-4903-a930-aa9a5f12c324"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:16:04.927038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:04.926965    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ae79f87-f000-4903-a930-aa9a5f12c324-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:16:04.928524 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:04.928499    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ae79f87-f000-4903-a930-aa9a5f12c324-maas-api-xks-ca" pod "8ae79f87-f000-4903-a930-aa9a5f12c324" (UID: "8ae79f87-f000-4903-a930-aa9a5f12c324"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:16:04.928637 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:04.928619    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae79f87-f000-4903-a930-aa9a5f12c324-kube-api-access-vx4jz" pod "8ae79f87-f000-4903-a930-aa9a5f12c324" (UID: "8ae79f87-f000-4903-a930-aa9a5f12c324"). InnerVolumeSpecName "kube-api-access-vx4jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:16:05.028539 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:05.028501    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ae79f87-f000-4903-a930-aa9a5f12c324-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:16:05.028539 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:05.028538    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx4jz\" (UniqueName: \"kubernetes.io/projected/8ae79f87-f000-4903-a930-aa9a5f12c324-kube-api-access-vx4jz\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:16:05.724371 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:05.724332    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"
Nov 02 12:16:05.724371 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:05.724346    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m" event={"ID":"8ae79f87-f000-4903-a930-aa9a5f12c324","Type":"ContainerDied","Data":"b5b37a50920a0503368e60d3144ba96128ef6cedd79d4824765c5ba68f594d1f"}
Nov 02 12:16:05.724567 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:05.724427    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5b37a50920a0503368e60d3144ba96128ef6cedd79d4824765c5ba68f594d1f"
Nov 02 12:16:07.792212 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.792175    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"]
Nov 02 12:16:07.792791 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.792778    4464 cpu_manager.go:414] "containerMap: removing container" podUID="8ae79f87-f000-4903-a930-aa9a5f12c324" containerName="revoke-keys"
Nov 02 12:16:07.792845 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.792792    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae79f87-f000-4903-a930-aa9a5f12c324" containerName="revoke-keys"
Nov 02 12:16:07.792903 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.792893    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae79f87-f000-4903-a930-aa9a5f12c324" containerName="revoke-keys"
Nov 02 12:16:07.796309 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.796291    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"
Nov 02 12:16:07.852609 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.852588    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:16:07.867570 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.867540    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53feb703-b468-42dc-aa83-8bca4e11a1c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e3a7ce-7dt8z\" (UID: \"53feb703-b468-42dc-aa83-8bca4e11a1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"
Nov 02 12:16:07.867697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.867662    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53feb703-b468-42dc-aa83-8bca4e11a1c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e3a7ce-7dt8z\" (UID: \"53feb703-b468-42dc-aa83-8bca4e11a1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"
Nov 02 12:16:07.867781 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.867755    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl5nb\" (UniqueName: \"kubernetes.io/projected/53feb703-b468-42dc-aa83-8bca4e11a1c9-kube-api-access-cl5nb\") pod \"maas-api-revoke-sub-12e3a7ce-7dt8z\" (UID: \"53feb703-b468-42dc-aa83-8bca4e11a1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"
Nov 02 12:16:07.903307 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.903284    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:16:07.903424 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.903306    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:16:07.903424 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.903315    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:16:07.907430 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.907409    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"]
Nov 02 12:16:07.968462 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.968428    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53feb703-b468-42dc-aa83-8bca4e11a1c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e3a7ce-7dt8z\" (UID: \"53feb703-b468-42dc-aa83-8bca4e11a1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"
Nov 02 12:16:07.968795 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.968541    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53feb703-b468-42dc-aa83-8bca4e11a1c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e3a7ce-7dt8z\" (UID: \"53feb703-b468-42dc-aa83-8bca4e11a1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"
Nov 02 12:16:07.968795 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.968588    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl5nb\" (UniqueName: \"kubernetes.io/projected/53feb703-b468-42dc-aa83-8bca4e11a1c9-kube-api-access-cl5nb\") pod \"maas-api-revoke-sub-12e3a7ce-7dt8z\" (UID: \"53feb703-b468-42dc-aa83-8bca4e11a1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"
Nov 02 12:16:07.969512 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.969485    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53feb703-b468-42dc-aa83-8bca4e11a1c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e3a7ce-7dt8z\" (UID: \"53feb703-b468-42dc-aa83-8bca4e11a1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"
Nov 02 12:16:07.971172 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:07.971143    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53feb703-b468-42dc-aa83-8bca4e11a1c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e3a7ce-7dt8z\" (UID: \"53feb703-b468-42dc-aa83-8bca4e11a1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"
Nov 02 12:16:08.030313 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:08.030288    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl5nb\" (UniqueName: \"kubernetes.io/projected/53feb703-b468-42dc-aa83-8bca4e11a1c9-kube-api-access-cl5nb\") pod \"maas-api-revoke-sub-12e3a7ce-7dt8z\" (UID: \"53feb703-b468-42dc-aa83-8bca4e11a1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"
Nov 02 12:16:08.109785 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:08.109725    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"
Nov 02 12:16:08.485304 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:16:08.485259    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53feb703_b468_42dc_aa83_8bca4e11a1c9.slice/crio-9215b685913dff16bb0216e522ed52e95dcb57fad6956636aeb8e7cc406338a6 WatchSource:0}: Error finding container 9215b685913dff16bb0216e522ed52e95dcb57fad6956636aeb8e7cc406338a6: Status 404 returned error can't find the container with id 9215b685913dff16bb0216e522ed52e95dcb57fad6956636aeb8e7cc406338a6
Nov 02 12:16:08.485896 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:08.485871    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"]
Nov 02 12:16:08.745852 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:08.745783    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z" event={"ID":"53feb703-b468-42dc-aa83-8bca4e11a1c9","Type":"ContainerStarted","Data":"9215b685913dff16bb0216e522ed52e95dcb57fad6956636aeb8e7cc406338a6"}
Nov 02 12:16:09.754514 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:09.754484    4464 generic.go:356] "Generic (PLEG): container finished" podID="53feb703-b468-42dc-aa83-8bca4e11a1c9" containerID="0b8a5418ee59300f0cdcbf8f48a0e19f7da73c260ccc5239b977a03d0c2aa5fb" exitCode=0
Nov 02 12:16:09.754884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:09.754561    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z" event={"ID":"53feb703-b468-42dc-aa83-8bca4e11a1c9","Type":"ContainerDied","Data":"0b8a5418ee59300f0cdcbf8f48a0e19f7da73c260ccc5239b977a03d0c2aa5fb"}
Nov 02 12:16:10.908669 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:10.908645    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"
Nov 02 12:16:11.006404 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:11.006371    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53feb703-b468-42dc-aa83-8bca4e11a1c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53feb703-b468-42dc-aa83-8bca4e11a1c9-maas-api-openshift-ca\") pod \"53feb703-b468-42dc-aa83-8bca4e11a1c9\" (UID: \"53feb703-b468-42dc-aa83-8bca4e11a1c9\") "
Nov 02 12:16:11.006572 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:11.006436    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53feb703-b468-42dc-aa83-8bca4e11a1c9-kube-api-access-cl5nb\" (UniqueName: \"kubernetes.io/projected/53feb703-b468-42dc-aa83-8bca4e11a1c9-kube-api-access-cl5nb\") pod \"53feb703-b468-42dc-aa83-8bca4e11a1c9\" (UID: \"53feb703-b468-42dc-aa83-8bca4e11a1c9\") "
Nov 02 12:16:11.006572 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:11.006489    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53feb703-b468-42dc-aa83-8bca4e11a1c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53feb703-b468-42dc-aa83-8bca4e11a1c9-maas-api-xks-ca\") pod \"53feb703-b468-42dc-aa83-8bca4e11a1c9\" (UID: \"53feb703-b468-42dc-aa83-8bca4e11a1c9\") "
Nov 02 12:16:11.006753 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:11.006731    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53feb703-b468-42dc-aa83-8bca4e11a1c9-maas-api-openshift-ca" pod "53feb703-b468-42dc-aa83-8bca4e11a1c9" (UID: "53feb703-b468-42dc-aa83-8bca4e11a1c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:16:11.008592 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:11.008560    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53feb703-b468-42dc-aa83-8bca4e11a1c9-maas-api-xks-ca" pod "53feb703-b468-42dc-aa83-8bca4e11a1c9" (UID: "53feb703-b468-42dc-aa83-8bca4e11a1c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:16:11.008703 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:11.008641    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53feb703-b468-42dc-aa83-8bca4e11a1c9-kube-api-access-cl5nb" pod "53feb703-b468-42dc-aa83-8bca4e11a1c9" (UID: "53feb703-b468-42dc-aa83-8bca4e11a1c9"). InnerVolumeSpecName "kube-api-access-cl5nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:16:11.108220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:11.108186    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53feb703-b468-42dc-aa83-8bca4e11a1c9-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:16:11.108220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:11.108212    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl5nb\" (UniqueName: \"kubernetes.io/projected/53feb703-b468-42dc-aa83-8bca4e11a1c9-kube-api-access-cl5nb\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:16:11.108220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:11.108223    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53feb703-b468-42dc-aa83-8bca4e11a1c9-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:16:11.704501 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:11.704471    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:16:11.706876 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:11.706853    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:16:11.723351 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:11.723322    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:16:11.725833 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:11.725811    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:16:11.768969 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:11.768946    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"
Nov 02 12:16:11.769104 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:11.769008    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z" event={"ID":"53feb703-b468-42dc-aa83-8bca4e11a1c9","Type":"ContainerDied","Data":"9215b685913dff16bb0216e522ed52e95dcb57fad6956636aeb8e7cc406338a6"}
Nov 02 12:16:11.769104 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:11.769084    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9215b685913dff16bb0216e522ed52e95dcb57fad6956636aeb8e7cc406338a6"
Nov 02 12:16:15.356868 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.356825    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"]
Nov 02 12:16:15.357629 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.357609    4464 cpu_manager.go:414] "containerMap: removing container" podUID="53feb703-b468-42dc-aa83-8bca4e11a1c9" containerName="revoke-keys"
Nov 02 12:16:15.357629 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.357629    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53feb703-b468-42dc-aa83-8bca4e11a1c9" containerName="revoke-keys"
Nov 02 12:16:15.357770 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.357759    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="53feb703-b468-42dc-aa83-8bca4e11a1c9" containerName="revoke-keys"
Nov 02 12:16:15.363543 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.363518    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.448009 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.441350    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-3a78b4-openshift-default-dockercfg-84mvc\""
Nov 02 12:16:15.475562 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.475516    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-workload-socket\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.475831 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.475804    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-podinfo\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.476166 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.476147    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-envoy\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.476392 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.476375    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-credential-socket\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.476507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.476494    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfm6r\" (UniqueName: \"kubernetes.io/projected/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-kube-api-access-mfm6r\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.476666 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.476632    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-data\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.477097 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.477075    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istiod-ca-cert\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.477333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.477317    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-token\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.477514 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.477488    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-workload-certs\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.612090 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.593302    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istiod-ca-cert\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.650001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.613599    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-token\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.690319 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.690250    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-workload-certs\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.691082 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.691062    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-workload-socket\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.691631 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.691614    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-podinfo\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.692163 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.692141    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-envoy\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.692420 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.692397    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-credential-socket\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.692649 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.692632    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfm6r\" (UniqueName: \"kubernetes.io/projected/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-kube-api-access-mfm6r\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.694189 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.694166    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-data\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.694808 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.694788    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-data\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.694930 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.691555    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-workload-socket\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.695861 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.695837    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-credential-socket\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.696000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.690962    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-workload-certs\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.746962 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.746920    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istiod-ca-cert\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.781325 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.780326    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-podinfo\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.795634 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.784130    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfm6r\" (UniqueName: \"kubernetes.io/projected/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-kube-api-access-mfm6r\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.800786 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.798905    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"]
Nov 02 12:16:15.805400 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.805322    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-envoy\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.815002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.810996    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-token\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:15.982770 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:15.982733    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:16.553251 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:16.550276    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:16:16.553251 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:16.550398    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:16:16.553251 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:16.550443    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:16:16.571002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:16.567910    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"]
Nov 02 12:16:16.899796 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:16.899756    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m" event={"ID":"86b728ba-2cb4-4e99-88b3-d6f0afd94f47","Type":"ContainerStarted","Data":"856408efffdfecedf42b7029cb3716cfcb97150f7cede905ef35532fa66b281f"}
Nov 02 12:16:17.951646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:17.949932    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m" event={"ID":"86b728ba-2cb4-4e99-88b3-d6f0afd94f47","Type":"ContainerStarted","Data":"f7b4fc003c3691b7cd85a3f2dd67b9a37e88606e966e6fec8798cbf4e8023b35"}
Nov 02 12:16:17.985559 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:17.985531    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:17.985873 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:17.985849    4464 patch_prober.go:29] interesting pod/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 02 12:16:17.986063 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:17.986036    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m" podUID="86b728ba-2cb4-4e99-88b3-d6f0afd94f47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 02 12:16:18.174000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:18.165944    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m" podStartSLOduration=3.165925279 podStartE2EDuration="3.165925279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:16:18.126371041 +0000 UTC m=+1206.690079530" watchObservedRunningTime="2026-11-02 12:16:18.165925279 +0000 UTC m=+1206.729633689"
Nov 02 12:16:18.460226 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:18.460175    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:16:18.460443 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:18.460251    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:16:18.987757 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:18.987727    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:19.345933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.345857    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"]
Nov 02 12:16:19.350208 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.350193    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"
Nov 02 12:16:19.404366 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.404332    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:16:19.404520 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.404452    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:16:19.404602 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.404577    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:16:19.404876 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.404851    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:16:19.417017 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.411494    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"]
Nov 02 12:16:19.463856 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.463831    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-457b8470-sp5p8\" (UID: \"0e14a726-d5db-43fa-9fa6-a697e7b6ec06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"
Nov 02 12:16:19.464072 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.464050    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-457b8470-sp5p8\" (UID: \"0e14a726-d5db-43fa-9fa6-a697e7b6ec06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"
Nov 02 12:16:19.464115 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.464101    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh2jk\" (UniqueName: \"kubernetes.io/projected/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-kube-api-access-sh2jk\") pod \"maas-api-revoke-sub-457b8470-sp5p8\" (UID: \"0e14a726-d5db-43fa-9fa6-a697e7b6ec06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"
Nov 02 12:16:19.564935 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.564903    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh2jk\" (UniqueName: \"kubernetes.io/projected/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-kube-api-access-sh2jk\") pod \"maas-api-revoke-sub-457b8470-sp5p8\" (UID: \"0e14a726-d5db-43fa-9fa6-a697e7b6ec06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"
Nov 02 12:16:19.565115 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.564999    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-457b8470-sp5p8\" (UID: \"0e14a726-d5db-43fa-9fa6-a697e7b6ec06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"
Nov 02 12:16:19.565209 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.565189    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-457b8470-sp5p8\" (UID: \"0e14a726-d5db-43fa-9fa6-a697e7b6ec06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"
Nov 02 12:16:19.565609 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.565588    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-457b8470-sp5p8\" (UID: \"0e14a726-d5db-43fa-9fa6-a697e7b6ec06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"
Nov 02 12:16:19.567493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.567473    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-457b8470-sp5p8\" (UID: \"0e14a726-d5db-43fa-9fa6-a697e7b6ec06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"
Nov 02 12:16:19.630635 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.630568    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh2jk\" (UniqueName: \"kubernetes.io/projected/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-kube-api-access-sh2jk\") pod \"maas-api-revoke-sub-457b8470-sp5p8\" (UID: \"0e14a726-d5db-43fa-9fa6-a697e7b6ec06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"
Nov 02 12:16:19.664052 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.664028    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"
Nov 02 12:16:19.964385 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.964314    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:19.965335 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:19.965313    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:16:20.041148 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:16:20.041114    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e14a726_d5db_43fa_9fa6_a697e7b6ec06.slice/crio-8608a368d773c56a1ec89218e654dd736dcf257f7bc687cf245535abb0efc5f1 WatchSource:0}: Error finding container 8608a368d773c56a1ec89218e654dd736dcf257f7bc687cf245535abb0efc5f1: Status 404 returned error can't find the container with id 8608a368d773c56a1ec89218e654dd736dcf257f7bc687cf245535abb0efc5f1
Nov 02 12:16:20.041495 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:20.041473    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"]
Nov 02 12:16:20.978915 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:20.978882    4464 generic.go:356] "Generic (PLEG): container finished" podID="0e14a726-d5db-43fa-9fa6-a697e7b6ec06" containerID="b68922ac35d776cc3e9105e01f84455d93acef8c4341cf121639f2d97b2ae861" exitCode=0
Nov 02 12:16:20.979084 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:20.978942    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8" event={"ID":"0e14a726-d5db-43fa-9fa6-a697e7b6ec06","Type":"ContainerDied","Data":"b68922ac35d776cc3e9105e01f84455d93acef8c4341cf121639f2d97b2ae861"}
Nov 02 12:16:20.979084 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:20.979024    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8" event={"ID":"0e14a726-d5db-43fa-9fa6-a697e7b6ec06","Type":"ContainerStarted","Data":"8608a368d773c56a1ec89218e654dd736dcf257f7bc687cf245535abb0efc5f1"}
Nov 02 12:16:22.131742 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:22.131718    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"
Nov 02 12:16:22.195460 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:22.195431    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-maas-api-xks-ca\") pod \"0e14a726-d5db-43fa-9fa6-a697e7b6ec06\" (UID: \"0e14a726-d5db-43fa-9fa6-a697e7b6ec06\") "
Nov 02 12:16:22.195599 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:22.195476    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-kube-api-access-sh2jk\" (UniqueName: \"kubernetes.io/projected/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-kube-api-access-sh2jk\") pod \"0e14a726-d5db-43fa-9fa6-a697e7b6ec06\" (UID: \"0e14a726-d5db-43fa-9fa6-a697e7b6ec06\") "
Nov 02 12:16:22.195599 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:22.195568    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-maas-api-openshift-ca\") pod \"0e14a726-d5db-43fa-9fa6-a697e7b6ec06\" (UID: \"0e14a726-d5db-43fa-9fa6-a697e7b6ec06\") "
Nov 02 12:16:22.195905 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:22.195877    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-maas-api-openshift-ca" pod "0e14a726-d5db-43fa-9fa6-a697e7b6ec06" (UID: "0e14a726-d5db-43fa-9fa6-a697e7b6ec06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:16:22.197615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:22.197595    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-maas-api-xks-ca" pod "0e14a726-d5db-43fa-9fa6-a697e7b6ec06" (UID: "0e14a726-d5db-43fa-9fa6-a697e7b6ec06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:16:22.197615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:22.197602    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-kube-api-access-sh2jk" pod "0e14a726-d5db-43fa-9fa6-a697e7b6ec06" (UID: "0e14a726-d5db-43fa-9fa6-a697e7b6ec06"). InnerVolumeSpecName "kube-api-access-sh2jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:16:22.296753 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:22.296681    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:16:22.296753 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:22.296709    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:16:22.296753 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:22.296731    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh2jk\" (UniqueName: \"kubernetes.io/projected/0e14a726-d5db-43fa-9fa6-a697e7b6ec06-kube-api-access-sh2jk\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:16:22.994452 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:22.994426    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"
Nov 02 12:16:22.994626 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:22.994467    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8" event={"ID":"0e14a726-d5db-43fa-9fa6-a697e7b6ec06","Type":"ContainerDied","Data":"8608a368d773c56a1ec89218e654dd736dcf257f7bc687cf245535abb0efc5f1"}
Nov 02 12:16:22.994626 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:22.994536    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8608a368d773c56a1ec89218e654dd736dcf257f7bc687cf245535abb0efc5f1"
Nov 02 12:16:24.733086 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:24.733002    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"]
Nov 02 12:16:24.733650 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:24.733632    4464 cpu_manager.go:414] "containerMap: removing container" podUID="0e14a726-d5db-43fa-9fa6-a697e7b6ec06" containerName="revoke-keys"
Nov 02 12:16:24.733694 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:24.733651    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e14a726-d5db-43fa-9fa6-a697e7b6ec06" containerName="revoke-keys"
Nov 02 12:16:24.733761 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:24.733751    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e14a726-d5db-43fa-9fa6-a697e7b6ec06" containerName="revoke-keys"
Nov 02 12:16:24.737342 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:24.737315    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"
Nov 02 12:16:24.841326 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:24.838867    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f8d75b7-8678-41a1-bbe0-f14a7107d9b8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9\" (UID: \"9f8d75b7-8678-41a1-bbe0-f14a7107d9b8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"
Nov 02 12:16:24.841707 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:24.841681    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xwdt\" (UniqueName: \"kubernetes.io/projected/9f8d75b7-8678-41a1-bbe0-f14a7107d9b8-kube-api-access-8xwdt\") pod \"payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9\" (UID: \"9f8d75b7-8678-41a1-bbe0-f14a7107d9b8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"
Nov 02 12:16:24.842341 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:24.842217    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"]
Nov 02 12:16:24.850414 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:24.850309    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-3a78b4-dockercfg-7qqs4\""
Nov 02 12:16:24.850414 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:24.850387    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-3a78b4\""
Nov 02 12:16:24.927061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:24.927030    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"]
Nov 02 12:16:24.931559 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:24.931539    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"
Nov 02 12:16:24.943419 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:24.943391    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f8d75b7-8678-41a1-bbe0-f14a7107d9b8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9\" (UID: \"9f8d75b7-8678-41a1-bbe0-f14a7107d9b8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"
Nov 02 12:16:24.943561 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:24.943451    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xwdt\" (UniqueName: \"kubernetes.io/projected/9f8d75b7-8678-41a1-bbe0-f14a7107d9b8-kube-api-access-8xwdt\") pod \"payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9\" (UID: \"9f8d75b7-8678-41a1-bbe0-f14a7107d9b8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"
Nov 02 12:16:24.944129 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:24.944110    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f8d75b7-8678-41a1-bbe0-f14a7107d9b8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9\" (UID: \"9f8d75b7-8678-41a1-bbe0-f14a7107d9b8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"
Nov 02 12:16:24.986194 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:24.986125    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"]
Nov 02 12:16:25.005373 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:25.005340    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xwdt\" (UniqueName: \"kubernetes.io/projected/9f8d75b7-8678-41a1-bbe0-f14a7107d9b8-kube-api-access-8xwdt\") pod \"payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9\" (UID: \"9f8d75b7-8678-41a1-bbe0-f14a7107d9b8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"
Nov 02 12:16:25.045220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:25.045189    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f65m\" (UniqueName: \"kubernetes.io/projected/ab236a0c-bd75-403b-8563-78617f65229e-kube-api-access-5f65m\") pod \"payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk\" (UID: \"ab236a0c-bd75-403b-8563-78617f65229e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"
Nov 02 12:16:25.045385 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:25.045367    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab236a0c-bd75-403b-8563-78617f65229e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk\" (UID: \"ab236a0c-bd75-403b-8563-78617f65229e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"
Nov 02 12:16:25.102724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:25.102697    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"
Nov 02 12:16:25.146152 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:25.146118    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab236a0c-bd75-403b-8563-78617f65229e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk\" (UID: \"ab236a0c-bd75-403b-8563-78617f65229e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"
Nov 02 12:16:25.146329 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:25.146243    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f65m\" (UniqueName: \"kubernetes.io/projected/ab236a0c-bd75-403b-8563-78617f65229e-kube-api-access-5f65m\") pod \"payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk\" (UID: \"ab236a0c-bd75-403b-8563-78617f65229e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"
Nov 02 12:16:25.146944 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:25.146922    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab236a0c-bd75-403b-8563-78617f65229e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk\" (UID: \"ab236a0c-bd75-403b-8563-78617f65229e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"
Nov 02 12:16:25.207937 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:25.207867    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f65m\" (UniqueName: \"kubernetes.io/projected/ab236a0c-bd75-403b-8563-78617f65229e-kube-api-access-5f65m\") pod \"payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk\" (UID: \"ab236a0c-bd75-403b-8563-78617f65229e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"
Nov 02 12:16:25.245840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:25.245769    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"
Nov 02 12:16:25.492150 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:16:25.492122    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8d75b7_8678_41a1_bbe0_f14a7107d9b8.slice/crio-a237236bc3caf10e048e5069bf3b2c23938d14c2ef79425b714068b712f5e936 WatchSource:0}: Error finding container a237236bc3caf10e048e5069bf3b2c23938d14c2ef79425b714068b712f5e936: Status 404 returned error can't find the container with id a237236bc3caf10e048e5069bf3b2c23938d14c2ef79425b714068b712f5e936
Nov 02 12:16:25.492254 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:25.492176    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"]
Nov 02 12:16:25.623333 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:16:25.623301    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab236a0c_bd75_403b_8563_78617f65229e.slice/crio-c0021257f06033f4d2007d83bb5e32bf730ba03e610e998550c48153d71fba32 WatchSource:0}: Error finding container c0021257f06033f4d2007d83bb5e32bf730ba03e610e998550c48153d71fba32: Status 404 returned error can't find the container with id c0021257f06033f4d2007d83bb5e32bf730ba03e610e998550c48153d71fba32
Nov 02 12:16:25.624085 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:25.624061    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"]
Nov 02 12:16:26.018058 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:26.018021    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9" event={"ID":"9f8d75b7-8678-41a1-bbe0-f14a7107d9b8","Type":"ContainerStarted","Data":"d845c72c2811d7f07d5a08b99cdd1690b80d7d69ba7c3384233e8f07ae4e5ec0"}
Nov 02 12:16:26.018466 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:26.018088    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9" event={"ID":"9f8d75b7-8678-41a1-bbe0-f14a7107d9b8","Type":"ContainerStarted","Data":"a237236bc3caf10e048e5069bf3b2c23938d14c2ef79425b714068b712f5e936"}
Nov 02 12:16:26.018466 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:26.018120    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"
Nov 02 12:16:26.019662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:26.019628    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk" event={"ID":"ab236a0c-bd75-403b-8563-78617f65229e","Type":"ContainerStarted","Data":"4e49ebae3549223ced3777afd27d7c246d8c979a7d774905e46b598679a3631e"}
Nov 02 12:16:26.019787 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:26.019673    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk" event={"ID":"ab236a0c-bd75-403b-8563-78617f65229e","Type":"ContainerStarted","Data":"c0021257f06033f4d2007d83bb5e32bf730ba03e610e998550c48153d71fba32"}
Nov 02 12:16:26.019787 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:26.019711    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"
Nov 02 12:16:26.140605 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:26.140552    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9" podStartSLOduration=2.140539787 podStartE2EDuration="2.140539787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:16:26.138514569 +0000 UTC m=+1214.702222977" watchObservedRunningTime="2026-11-02 12:16:26.140539787 +0000 UTC m=+1214.704248211"
Nov 02 12:16:28.292656 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:16:28.292614    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:16:34.977230 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:34.977176    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk" podStartSLOduration=10.97716351 podStartE2EDuration="10.97716351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:16:26.284458007 +0000 UTC m=+1214.848166414" watchObservedRunningTime="2026-11-02 12:16:34.97716351 +0000 UTC m=+1223.540871916"
Nov 02 12:16:34.980063 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:34.980042    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"]
Nov 02 12:16:34.984439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:34.984423    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"
Nov 02 12:16:35.038191 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.038145    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:16:35.088790 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.088763    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:16:35.088790 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.088792    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:16:35.089030 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.088798    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:16:35.093206 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.093181    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"]
Nov 02 12:16:35.097274 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.097257    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"]
Nov 02 12:16:35.100796 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.100777    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc3a5d26-smbzj"]
Nov 02 12:16:35.104070 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.104039    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kw8t\" (UniqueName: \"kubernetes.io/projected/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-kube-api-access-6kw8t\") pod \"maas-api-revoke-sub-63e8a4ce-fms2n\" (UID: \"a3e13b9b-3c31-43f2-abab-ddab6183e2fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"
Nov 02 12:16:35.104184 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.104167    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63e8a4ce-fms2n\" (UID: \"a3e13b9b-3c31-43f2-abab-ddab6183e2fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"
Nov 02 12:16:35.104223 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.104193    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63e8a4ce-fms2n\" (UID: \"a3e13b9b-3c31-43f2-abab-ddab6183e2fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"
Nov 02 12:16:35.205818 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.205773    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kw8t\" (UniqueName: \"kubernetes.io/projected/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-kube-api-access-6kw8t\") pod \"maas-api-revoke-sub-63e8a4ce-fms2n\" (UID: \"a3e13b9b-3c31-43f2-abab-ddab6183e2fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"
Nov 02 12:16:35.206048 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.205870    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63e8a4ce-fms2n\" (UID: \"a3e13b9b-3c31-43f2-abab-ddab6183e2fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"
Nov 02 12:16:35.206048 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.205887    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63e8a4ce-fms2n\" (UID: \"a3e13b9b-3c31-43f2-abab-ddab6183e2fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"
Nov 02 12:16:35.206537 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.206518    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63e8a4ce-fms2n\" (UID: \"a3e13b9b-3c31-43f2-abab-ddab6183e2fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"
Nov 02 12:16:35.208200 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.208181    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63e8a4ce-fms2n\" (UID: \"a3e13b9b-3c31-43f2-abab-ddab6183e2fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"
Nov 02 12:16:35.268424 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.268356    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kw8t\" (UniqueName: \"kubernetes.io/projected/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-kube-api-access-6kw8t\") pod \"maas-api-revoke-sub-63e8a4ce-fms2n\" (UID: \"a3e13b9b-3c31-43f2-abab-ddab6183e2fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"
Nov 02 12:16:35.298485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.298459    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"
Nov 02 12:16:35.583123 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.583042    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="912ce582-7325-436e-bf7b-6d3b98034bb7" path="/var/lib/kubelet/pods/912ce582-7325-436e-bf7b-6d3b98034bb7/volumes"
Nov 02 12:16:35.680604 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:16:35.680569    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e13b9b_3c31_43f2_abab_ddab6183e2fe.slice/crio-8ac73ed87c91dd0fb4dc4b711f4b30bebef6b2a6fd46f6e5499b11d5a3ddf871 WatchSource:0}: Error finding container 8ac73ed87c91dd0fb4dc4b711f4b30bebef6b2a6fd46f6e5499b11d5a3ddf871: Status 404 returned error can't find the container with id 8ac73ed87c91dd0fb4dc4b711f4b30bebef6b2a6fd46f6e5499b11d5a3ddf871
Nov 02 12:16:35.681262 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:35.681239    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"]
Nov 02 12:16:36.093475 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:36.093443    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n" event={"ID":"a3e13b9b-3c31-43f2-abab-ddab6183e2fe","Type":"ContainerStarted","Data":"765c7696a84f757f0ce867bb1bcb2704ea5db0bb15227d6030a5b70926b44e44"}
Nov 02 12:16:36.093823 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:36.093499    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n" event={"ID":"a3e13b9b-3c31-43f2-abab-ddab6183e2fe","Type":"ContainerStarted","Data":"8ac73ed87c91dd0fb4dc4b711f4b30bebef6b2a6fd46f6e5499b11d5a3ddf871"}
Nov 02 12:16:36.212189 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:36.212092    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n" podStartSLOduration=2.212076583 podStartE2EDuration="2.212076583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:16:36.210736553 +0000 UTC m=+1224.774444962" watchObservedRunningTime="2026-11-02 12:16:36.212076583 +0000 UTC m=+1224.775785043"
Nov 02 12:16:37.029760 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:37.029724    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"
Nov 02 12:16:37.029952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:37.029815    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"
Nov 02 12:16:37.101998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:37.101957    4464 generic.go:356] "Generic (PLEG): container finished" podID="a3e13b9b-3c31-43f2-abab-ddab6183e2fe" containerID="765c7696a84f757f0ce867bb1bcb2704ea5db0bb15227d6030a5b70926b44e44" exitCode=0
Nov 02 12:16:37.102377 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:37.102041    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n" event={"ID":"a3e13b9b-3c31-43f2-abab-ddab6183e2fe","Type":"ContainerDied","Data":"765c7696a84f757f0ce867bb1bcb2704ea5db0bb15227d6030a5b70926b44e44"}
Nov 02 12:16:37.120411 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:37.120378    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:16:39.340150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:39.340110    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"
Nov 02 12:16:39.472818 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:39.472764    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-maas-api-openshift-ca" pod "a3e13b9b-3c31-43f2-abab-ddab6183e2fe" (UID: "a3e13b9b-3c31-43f2-abab-ddab6183e2fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:16:39.473056 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:39.472353    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-maas-api-openshift-ca\") pod \"a3e13b9b-3c31-43f2-abab-ddab6183e2fe\" (UID: \"a3e13b9b-3c31-43f2-abab-ddab6183e2fe\") "
Nov 02 12:16:39.473239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:39.473211    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-maas-api-xks-ca\") pod \"a3e13b9b-3c31-43f2-abab-ddab6183e2fe\" (UID: \"a3e13b9b-3c31-43f2-abab-ddab6183e2fe\") "
Nov 02 12:16:39.473451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:39.473433    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-kube-api-access-6kw8t\" (UniqueName: \"kubernetes.io/projected/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-kube-api-access-6kw8t\") pod \"a3e13b9b-3c31-43f2-abab-ddab6183e2fe\" (UID: \"a3e13b9b-3c31-43f2-abab-ddab6183e2fe\") "
Nov 02 12:16:39.482333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:39.482297    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:16:39.532191 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:39.532132    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-maas-api-xks-ca" pod "a3e13b9b-3c31-43f2-abab-ddab6183e2fe" (UID: "a3e13b9b-3c31-43f2-abab-ddab6183e2fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:16:39.568996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:39.549069    4464 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv" podUID="d829dab9-d274-485e-bef3-a29c5ddfff22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:16:39.743051 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:39.722914    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:16:39.851611 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:39.850896    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-kube-api-access-6kw8t" pod "a3e13b9b-3c31-43f2-abab-ddab6183e2fe" (UID: "a3e13b9b-3c31-43f2-abab-ddab6183e2fe"). InnerVolumeSpecName "kube-api-access-6kw8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:16:39.891374 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:39.890051    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf" podUID="ff6c0679-4d25-499c-809d-4e22187c23ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Nov 02 12:16:39.943647 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:39.942229    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kw8t\" (UniqueName: \"kubernetes.io/projected/a3e13b9b-3c31-43f2-abab-ddab6183e2fe-kube-api-access-6kw8t\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:16:39.958641 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:39.958606    4464 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72" podUID="df859e10-37e4-4d2e-a1d5-931adacbaaff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:16:40.324723 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:40.324692    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"
Nov 02 12:16:40.324903 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:40.324719    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n" event={"ID":"a3e13b9b-3c31-43f2-abab-ddab6183e2fe","Type":"ContainerDied","Data":"8ac73ed87c91dd0fb4dc4b711f4b30bebef6b2a6fd46f6e5499b11d5a3ddf871"}
Nov 02 12:16:40.324903 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:40.324786    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ac73ed87c91dd0fb4dc4b711f4b30bebef6b2a6fd46f6e5499b11d5a3ddf871"
Nov 02 12:16:50.433005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.430808    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"]
Nov 02 12:16:50.433005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.432309    4464 cpu_manager.go:414] "containerMap: removing container" podUID="a3e13b9b-3c31-43f2-abab-ddab6183e2fe" containerName="revoke-keys"
Nov 02 12:16:50.433005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.432331    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e13b9b-3c31-43f2-abab-ddab6183e2fe" containerName="revoke-keys"
Nov 02 12:16:50.433005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.432675    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3e13b9b-3c31-43f2-abab-ddab6183e2fe" containerName="revoke-keys"
Nov 02 12:16:50.438240 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.438214    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"
Nov 02 12:16:50.494070 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.494046    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"]
Nov 02 12:16:50.539681 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.539651    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:16:50.539681 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.539675    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:16:50.539837 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.539684    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:16:50.546168 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.546146    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:16:50.573418 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.573390    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd1a779-5e7b-45df-890c-b759b3b17f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-370e6bbb-zk8wh\" (UID: \"bdd1a779-5e7b-45df-890c-b759b3b17f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"
Nov 02 12:16:50.573509 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.573440    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd1a779-5e7b-45df-890c-b759b3b17f3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-370e6bbb-zk8wh\" (UID: \"bdd1a779-5e7b-45df-890c-b759b3b17f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"
Nov 02 12:16:50.573595 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.573577    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6gv4\" (UniqueName: \"kubernetes.io/projected/bdd1a779-5e7b-45df-890c-b759b3b17f3c-kube-api-access-r6gv4\") pod \"maas-api-revoke-sub-370e6bbb-zk8wh\" (UID: \"bdd1a779-5e7b-45df-890c-b759b3b17f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"
Nov 02 12:16:50.674939 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.674913    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6gv4\" (UniqueName: \"kubernetes.io/projected/bdd1a779-5e7b-45df-890c-b759b3b17f3c-kube-api-access-r6gv4\") pod \"maas-api-revoke-sub-370e6bbb-zk8wh\" (UID: \"bdd1a779-5e7b-45df-890c-b759b3b17f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"
Nov 02 12:16:50.675089 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.675017    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd1a779-5e7b-45df-890c-b759b3b17f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-370e6bbb-zk8wh\" (UID: \"bdd1a779-5e7b-45df-890c-b759b3b17f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"
Nov 02 12:16:50.675089 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.675052    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd1a779-5e7b-45df-890c-b759b3b17f3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-370e6bbb-zk8wh\" (UID: \"bdd1a779-5e7b-45df-890c-b759b3b17f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"
Nov 02 12:16:50.675620 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.675599    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd1a779-5e7b-45df-890c-b759b3b17f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-370e6bbb-zk8wh\" (UID: \"bdd1a779-5e7b-45df-890c-b759b3b17f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"
Nov 02 12:16:50.677374 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.677354    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd1a779-5e7b-45df-890c-b759b3b17f3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-370e6bbb-zk8wh\" (UID: \"bdd1a779-5e7b-45df-890c-b759b3b17f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"
Nov 02 12:16:50.736680 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.736654    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6gv4\" (UniqueName: \"kubernetes.io/projected/bdd1a779-5e7b-45df-890c-b759b3b17f3c-kube-api-access-r6gv4\") pod \"maas-api-revoke-sub-370e6bbb-zk8wh\" (UID: \"bdd1a779-5e7b-45df-890c-b759b3b17f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"
Nov 02 12:16:50.751353 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:50.751333    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"
Nov 02 12:16:51.129457 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:16:51.129425    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd1a779_5e7b_45df_890c_b759b3b17f3c.slice/crio-078cf1be0f9d5184bab23f212c93498a342c473de3f7b98dfa098f93d217a3d8 WatchSource:0}: Error finding container 078cf1be0f9d5184bab23f212c93498a342c473de3f7b98dfa098f93d217a3d8: Status 404 returned error can't find the container with id 078cf1be0f9d5184bab23f212c93498a342c473de3f7b98dfa098f93d217a3d8
Nov 02 12:16:51.130119 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:51.130096    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"]
Nov 02 12:16:51.404216 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:51.404150    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh" event={"ID":"bdd1a779-5e7b-45df-890c-b759b3b17f3c","Type":"ContainerStarted","Data":"078cf1be0f9d5184bab23f212c93498a342c473de3f7b98dfa098f93d217a3d8"}
Nov 02 12:16:52.413683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:52.413600    4464 generic.go:356] "Generic (PLEG): container finished" podID="bdd1a779-5e7b-45df-890c-b759b3b17f3c" containerID="5bff59ff8da5bd7be4ce8d1364bfa0882692fdfdd62a5d081a114edac3bc2989" exitCode=0
Nov 02 12:16:52.414064 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:52.413682    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh" event={"ID":"bdd1a779-5e7b-45df-890c-b759b3b17f3c","Type":"ContainerDied","Data":"5bff59ff8da5bd7be4ce8d1364bfa0882692fdfdd62a5d081a114edac3bc2989"}
Nov 02 12:16:53.572246 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:53.572221    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"
Nov 02 12:16:53.710012 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:53.709898    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd1a779-5e7b-45df-890c-b759b3b17f3c-kube-api-access-r6gv4\" (UniqueName: \"kubernetes.io/projected/bdd1a779-5e7b-45df-890c-b759b3b17f3c-kube-api-access-r6gv4\") pod \"bdd1a779-5e7b-45df-890c-b759b3b17f3c\" (UID: \"bdd1a779-5e7b-45df-890c-b759b3b17f3c\") "
Nov 02 12:16:53.710186 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:53.710071    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdd1a779-5e7b-45df-890c-b759b3b17f3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd1a779-5e7b-45df-890c-b759b3b17f3c-maas-api-xks-ca\") pod \"bdd1a779-5e7b-45df-890c-b759b3b17f3c\" (UID: \"bdd1a779-5e7b-45df-890c-b759b3b17f3c\") "
Nov 02 12:16:53.710254 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:53.710199    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd1a779-5e7b-45df-890c-b759b3b17f3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd1a779-5e7b-45df-890c-b759b3b17f3c-maas-api-openshift-ca\") pod \"bdd1a779-5e7b-45df-890c-b759b3b17f3c\" (UID: \"bdd1a779-5e7b-45df-890c-b759b3b17f3c\") "
Nov 02 12:16:53.710639 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:53.710604    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd1a779-5e7b-45df-890c-b759b3b17f3c-maas-api-openshift-ca" pod "bdd1a779-5e7b-45df-890c-b759b3b17f3c" (UID: "bdd1a779-5e7b-45df-890c-b759b3b17f3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:16:53.712653 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:53.712618    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd1a779-5e7b-45df-890c-b759b3b17f3c-maas-api-xks-ca" pod "bdd1a779-5e7b-45df-890c-b759b3b17f3c" (UID: "bdd1a779-5e7b-45df-890c-b759b3b17f3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:16:53.712653 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:53.712623    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd1a779-5e7b-45df-890c-b759b3b17f3c-kube-api-access-r6gv4" pod "bdd1a779-5e7b-45df-890c-b759b3b17f3c" (UID: "bdd1a779-5e7b-45df-890c-b759b3b17f3c"). InnerVolumeSpecName "kube-api-access-r6gv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:16:53.812275 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:53.812244    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd1a779-5e7b-45df-890c-b759b3b17f3c-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:16:53.812275 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:53.812270    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6gv4\" (UniqueName: \"kubernetes.io/projected/bdd1a779-5e7b-45df-890c-b759b3b17f3c-kube-api-access-r6gv4\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:16:53.812275 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:53.812280    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd1a779-5e7b-45df-890c-b759b3b17f3c-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:16:54.073145 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:54.073024    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"]
Nov 02 12:16:54.082774 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:54.082738    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf582f8-72g4l"]
Nov 02 12:16:54.500999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:54.489767    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh" event={"ID":"bdd1a779-5e7b-45df-890c-b759b3b17f3c","Type":"ContainerDied","Data":"078cf1be0f9d5184bab23f212c93498a342c473de3f7b98dfa098f93d217a3d8"}
Nov 02 12:16:54.500999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:54.489882    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="078cf1be0f9d5184bab23f212c93498a342c473de3f7b98dfa098f93d217a3d8"
Nov 02 12:16:54.500999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:54.489951    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"
Nov 02 12:16:55.771999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:55.771816    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d780d2f-1755-4056-a811-72279983a394" path="/var/lib/kubelet/pods/6d780d2f-1755-4056-a811-72279983a394/volumes"
Nov 02 12:16:55.921212 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:16:55.921136    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf" podUID="ff6c0679-4d25-499c-809d-4e22187c23ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:16:58.293253 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:16:58.293220    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:17:05.958523 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:05.958485    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"]
Nov 02 12:17:05.959409 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:05.959385    4464 cpu_manager.go:414] "containerMap: removing container" podUID="bdd1a779-5e7b-45df-890c-b759b3b17f3c" containerName="revoke-keys"
Nov 02 12:17:05.959409 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:05.959409    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd1a779-5e7b-45df-890c-b759b3b17f3c" containerName="revoke-keys"
Nov 02 12:17:05.959605 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:05.959592    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd1a779-5e7b-45df-890c-b759b3b17f3c" containerName="revoke-keys"
Nov 02 12:17:05.963276 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:05.963253    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"
Nov 02 12:17:06.019344 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:06.019313    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"]
Nov 02 12:17:06.047361 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:06.047332    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8c8665-1f21-4d88-9119-6d4826b40a8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba9b8cf1-htvhp\" (UID: \"0f8c8665-1f21-4d88-9119-6d4826b40a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"
Nov 02 12:17:06.047486 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:06.047461    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8c8665-1f21-4d88-9119-6d4826b40a8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba9b8cf1-htvhp\" (UID: \"0f8c8665-1f21-4d88-9119-6d4826b40a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"
Nov 02 12:17:06.047527 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:06.047518    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6mm6\" (UniqueName: \"kubernetes.io/projected/0f8c8665-1f21-4d88-9119-6d4826b40a8d-kube-api-access-g6mm6\") pod \"maas-api-revoke-sub-ba9b8cf1-htvhp\" (UID: \"0f8c8665-1f21-4d88-9119-6d4826b40a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"
Nov 02 12:17:06.065137 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:06.065117    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:17:06.065275 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:06.065198    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:17:06.067145 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:06.067126    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:17:06.067217 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:06.067133    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:17:06.148964 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:06.148939    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8c8665-1f21-4d88-9119-6d4826b40a8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba9b8cf1-htvhp\" (UID: \"0f8c8665-1f21-4d88-9119-6d4826b40a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"
Nov 02 12:17:06.149102 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:06.149071    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8c8665-1f21-4d88-9119-6d4826b40a8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba9b8cf1-htvhp\" (UID: \"0f8c8665-1f21-4d88-9119-6d4826b40a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"
Nov 02 12:17:06.149156 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:06.149119    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6mm6\" (UniqueName: \"kubernetes.io/projected/0f8c8665-1f21-4d88-9119-6d4826b40a8d-kube-api-access-g6mm6\") pod \"maas-api-revoke-sub-ba9b8cf1-htvhp\" (UID: \"0f8c8665-1f21-4d88-9119-6d4826b40a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"
Nov 02 12:17:06.149700 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:06.149677    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8c8665-1f21-4d88-9119-6d4826b40a8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba9b8cf1-htvhp\" (UID: \"0f8c8665-1f21-4d88-9119-6d4826b40a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"
Nov 02 12:17:06.151347 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:06.151326    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8c8665-1f21-4d88-9119-6d4826b40a8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba9b8cf1-htvhp\" (UID: \"0f8c8665-1f21-4d88-9119-6d4826b40a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"
Nov 02 12:17:06.210919 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:06.210865    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6mm6\" (UniqueName: \"kubernetes.io/projected/0f8c8665-1f21-4d88-9119-6d4826b40a8d-kube-api-access-g6mm6\") pod \"maas-api-revoke-sub-ba9b8cf1-htvhp\" (UID: \"0f8c8665-1f21-4d88-9119-6d4826b40a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"
Nov 02 12:17:06.277710 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:06.277685    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"
Nov 02 12:17:06.656350 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:06.656321    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"]
Nov 02 12:17:06.657080 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:17:06.657058    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8c8665_1f21_4d88_9119_6d4826b40a8d.slice/crio-2f8d6dceb44ae8f7280d6a9a49c7cd9233c2b8a872aecc0f7605af7ab43ca29e WatchSource:0}: Error finding container 2f8d6dceb44ae8f7280d6a9a49c7cd9233c2b8a872aecc0f7605af7ab43ca29e: Status 404 returned error can't find the container with id 2f8d6dceb44ae8f7280d6a9a49c7cd9233c2b8a872aecc0f7605af7ab43ca29e
Nov 02 12:17:06.889250 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:06.887637    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp" event={"ID":"0f8c8665-1f21-4d88-9119-6d4826b40a8d","Type":"ContainerStarted","Data":"2f8d6dceb44ae8f7280d6a9a49c7cd9233c2b8a872aecc0f7605af7ab43ca29e"}
Nov 02 12:17:07.896747 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:07.896666    4464 generic.go:356] "Generic (PLEG): container finished" podID="0f8c8665-1f21-4d88-9119-6d4826b40a8d" containerID="9129fa7db4e9d9af2ac358bd56973cf84547583b2b5d02789ac136d6a76aadc5" exitCode=0
Nov 02 12:17:07.896747 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:07.896719    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp" event={"ID":"0f8c8665-1f21-4d88-9119-6d4826b40a8d","Type":"ContainerDied","Data":"9129fa7db4e9d9af2ac358bd56973cf84547583b2b5d02789ac136d6a76aadc5"}
Nov 02 12:17:09.086621 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:09.086586    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"
Nov 02 12:17:09.196516 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:09.196447    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f8c8665-1f21-4d88-9119-6d4826b40a8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8c8665-1f21-4d88-9119-6d4826b40a8d-maas-api-openshift-ca\") pod \"0f8c8665-1f21-4d88-9119-6d4826b40a8d\" (UID: \"0f8c8665-1f21-4d88-9119-6d4826b40a8d\") "
Nov 02 12:17:09.196652 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:09.196577    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8c8665-1f21-4d88-9119-6d4826b40a8d-kube-api-access-g6mm6\" (UniqueName: \"kubernetes.io/projected/0f8c8665-1f21-4d88-9119-6d4826b40a8d-kube-api-access-g6mm6\") pod \"0f8c8665-1f21-4d88-9119-6d4826b40a8d\" (UID: \"0f8c8665-1f21-4d88-9119-6d4826b40a8d\") "
Nov 02 12:17:09.196652 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:09.196613    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f8c8665-1f21-4d88-9119-6d4826b40a8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8c8665-1f21-4d88-9119-6d4826b40a8d-maas-api-xks-ca\") pod \"0f8c8665-1f21-4d88-9119-6d4826b40a8d\" (UID: \"0f8c8665-1f21-4d88-9119-6d4826b40a8d\") "
Nov 02 12:17:09.196784 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:09.196762    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8c8665-1f21-4d88-9119-6d4826b40a8d-maas-api-openshift-ca" pod "0f8c8665-1f21-4d88-9119-6d4826b40a8d" (UID: "0f8c8665-1f21-4d88-9119-6d4826b40a8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:17:09.197016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:09.196998    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8c8665-1f21-4d88-9119-6d4826b40a8d-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:09.198543 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:09.198513    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8c8665-1f21-4d88-9119-6d4826b40a8d-kube-api-access-g6mm6" pod "0f8c8665-1f21-4d88-9119-6d4826b40a8d" (UID: "0f8c8665-1f21-4d88-9119-6d4826b40a8d"). InnerVolumeSpecName "kube-api-access-g6mm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:17:09.198652 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:09.198578    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f8c8665-1f21-4d88-9119-6d4826b40a8d-maas-api-xks-ca" pod "0f8c8665-1f21-4d88-9119-6d4826b40a8d" (UID: "0f8c8665-1f21-4d88-9119-6d4826b40a8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:17:09.297816 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:09.297779    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6mm6\" (UniqueName: \"kubernetes.io/projected/0f8c8665-1f21-4d88-9119-6d4826b40a8d-kube-api-access-g6mm6\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:09.297816 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:09.297810    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8c8665-1f21-4d88-9119-6d4826b40a8d-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:09.912214 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:09.912134    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"
Nov 02 12:17:09.912214 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:09.912147    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp" event={"ID":"0f8c8665-1f21-4d88-9119-6d4826b40a8d","Type":"ContainerDied","Data":"2f8d6dceb44ae8f7280d6a9a49c7cd9233c2b8a872aecc0f7605af7ab43ca29e"}
Nov 02 12:17:09.912214 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:09.912189    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f8d6dceb44ae8f7280d6a9a49c7cd9233c2b8a872aecc0f7605af7ab43ca29e"
Nov 02 12:17:10.909567 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:10.909537    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:17:12.053443 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:12.053411    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"]
Nov 02 12:17:12.106772 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:12.106737    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-040dbd72-rwlsd"]
Nov 02 12:17:12.831881 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:12.831832    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"]
Nov 02 12:17:12.833090 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:12.833062    4464 cpu_manager.go:414] "containerMap: removing container" podUID="0f8c8665-1f21-4d88-9119-6d4826b40a8d" containerName="revoke-keys"
Nov 02 12:17:12.833245 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:12.833229    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8c8665-1f21-4d88-9119-6d4826b40a8d" containerName="revoke-keys"
Nov 02 12:17:12.833562 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:12.833543    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8c8665-1f21-4d88-9119-6d4826b40a8d" containerName="revoke-keys"
Nov 02 12:17:12.841192 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:12.841166    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:12.957158 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:12.957116    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"]
Nov 02 12:17:12.974348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:12.974318    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-036f88-openshift-default-dockercfg-gfh5b\""
Nov 02 12:17:13.052485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.051827    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-credential-socket\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.052485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.051888    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-envoy\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.052485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.051961    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-workload-certs\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.052485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.052021    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-token\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.052485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.052048    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-podinfo\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.052485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.052075    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2hj2\" (UniqueName: \"kubernetes.io/projected/f9d165c8-5462-4659-9ca7-65df7b86f133-kube-api-access-b2hj2\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.052485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.052136    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-data\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.052485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.052174    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-workload-socket\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.052485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.052254    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d165c8-5462-4659-9ca7-65df7b86f133-istiod-ca-cert\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.153679 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.153592    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d165c8-5462-4659-9ca7-65df7b86f133-istiod-ca-cert\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.154162 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.153685    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-credential-socket\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.154162 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.153713    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-envoy\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.154162 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.153757    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-workload-certs\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.154162 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.153780    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-token\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.154162 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.153801    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-podinfo\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.154162 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.153829    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2hj2\" (UniqueName: \"kubernetes.io/projected/f9d165c8-5462-4659-9ca7-65df7b86f133-kube-api-access-b2hj2\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.154162 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.153883    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-data\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.154162 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.153918    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-workload-socket\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.154162 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.154148    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-credential-socket\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.154507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.154185    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-workload-certs\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.154507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.154361    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-workload-socket\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.154507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.154434    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d165c8-5462-4659-9ca7-65df7b86f133-istiod-ca-cert\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.154507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.154440    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-data\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.156310 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.156285    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-envoy\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.156424 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.156400    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-podinfo\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.223086 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.223011    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2hj2\" (UniqueName: \"kubernetes.io/projected/f9d165c8-5462-4659-9ca7-65df7b86f133-kube-api-access-b2hj2\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.230488 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.230461    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-token\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.457258 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.457169    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:13.627510 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:13.627474    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c8dc80-9a57-4624-be9b-8a73c1141dff" path="/var/lib/kubelet/pods/47c8dc80-9a57-4624-be9b-8a73c1141dff/volumes"
Nov 02 12:17:14.025617 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:17:14.025590    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d165c8_5462_4659_9ca7_65df7b86f133.slice/crio-502480a2483a6b257a7c0a3f7927c72939d5f731af26b5648cbc5700ffe70a69 WatchSource:0}: Error finding container 502480a2483a6b257a7c0a3f7927c72939d5f731af26b5648cbc5700ffe70a69: Status 404 returned error can't find the container with id 502480a2483a6b257a7c0a3f7927c72939d5f731af26b5648cbc5700ffe70a69
Nov 02 12:17:14.026258 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:14.026235    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"]
Nov 02 12:17:14.028696 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:14.028664    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:17:14.028777 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:14.028727    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:17:14.028777 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:14.028756    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:17:14.999513 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:14.999472    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd" event={"ID":"f9d165c8-5462-4659-9ca7-65df7b86f133","Type":"ContainerStarted","Data":"9a625e8af0f5676afcea662460c267046227d796a0a02c74be4c8794dc1d8363"}
Nov 02 12:17:14.999923 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:14.999567    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd" event={"ID":"f9d165c8-5462-4659-9ca7-65df7b86f133","Type":"ContainerStarted","Data":"502480a2483a6b257a7c0a3f7927c72939d5f731af26b5648cbc5700ffe70a69"}
Nov 02 12:17:15.224927 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:15.224870    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd" podStartSLOduration=3.224856357 podStartE2EDuration="3.224856357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:17:15.178616369 +0000 UTC m=+1263.742324779" watchObservedRunningTime="2026-11-02 12:17:15.224856357 +0000 UTC m=+1263.788564843"
Nov 02 12:17:15.458178 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:15.458094    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:15.484171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:15.484132    4464 patch_prober.go:29] interesting pod/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 02 12:17:15.484361 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:15.484209    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd" podUID="f9d165c8-5462-4659-9ca7-65df7b86f133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 02 12:17:16.458290 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:16.458257    4464 patch_prober.go:29] interesting pod/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 02 12:17:16.458709 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:16.458323    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd" podUID="f9d165c8-5462-4659-9ca7-65df7b86f133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 02 12:17:17.458177 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:17.458145    4464 patch_prober.go:29] interesting pod/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 02 12:17:17.458359 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:17.458212    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd" podUID="f9d165c8-5462-4659-9ca7-65df7b86f133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 02 12:17:18.457710 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:18.457669    4464 patch_prober.go:29] interesting pod/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 02 12:17:18.457854 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:18.457733    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd" podUID="f9d165c8-5462-4659-9ca7-65df7b86f133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 02 12:17:19.457488 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:19.457454    4464 patch_prober.go:29] interesting pod/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 02 12:17:19.457885 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:19.457514    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd" podUID="f9d165c8-5462-4659-9ca7-65df7b86f133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 02 12:17:20.458466 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:20.458431    4464 patch_prober.go:29] interesting pod/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 02 12:17:20.458924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:20.458501    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd" podUID="f9d165c8-5462-4659-9ca7-65df7b86f133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 02 12:17:21.462183 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:21.462153    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:21.462669 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:21.462582    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:21.463153 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:21.463137    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:17:21.984054 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:21.984021    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"]
Nov 02 12:17:21.988358 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:21.988338    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"
Nov 02 12:17:22.043560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:22.043497    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:17:22.043560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:22.043524    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:17:22.043560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:22.043509    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:17:22.043849 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:22.043823    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:17:22.046726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:22.046703    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"]
Nov 02 12:17:22.053605 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:22.053577    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-265722de-ff6lg\" (UID: \"c5223feb-e5f0-4e64-95a9-c7f4235e99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"
Nov 02 12:17:22.053883 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:22.053854    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-265722de-ff6lg\" (UID: \"c5223feb-e5f0-4e64-95a9-c7f4235e99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"
Nov 02 12:17:22.054023 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:22.053964    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkjkf\" (UniqueName: \"kubernetes.io/projected/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-kube-api-access-nkjkf\") pod \"maas-api-revoke-sub-265722de-ff6lg\" (UID: \"c5223feb-e5f0-4e64-95a9-c7f4235e99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"
Nov 02 12:17:22.155460 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:22.155424    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-265722de-ff6lg\" (UID: \"c5223feb-e5f0-4e64-95a9-c7f4235e99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"
Nov 02 12:17:22.155644 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:22.155520    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkjkf\" (UniqueName: \"kubernetes.io/projected/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-kube-api-access-nkjkf\") pod \"maas-api-revoke-sub-265722de-ff6lg\" (UID: \"c5223feb-e5f0-4e64-95a9-c7f4235e99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"
Nov 02 12:17:22.155644 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:22.155561    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-265722de-ff6lg\" (UID: \"c5223feb-e5f0-4e64-95a9-c7f4235e99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"
Nov 02 12:17:22.156259 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:22.156233    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-265722de-ff6lg\" (UID: \"c5223feb-e5f0-4e64-95a9-c7f4235e99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"
Nov 02 12:17:22.157730 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:22.157706    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-265722de-ff6lg\" (UID: \"c5223feb-e5f0-4e64-95a9-c7f4235e99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"
Nov 02 12:17:22.217699 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:22.217671    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkjkf\" (UniqueName: \"kubernetes.io/projected/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-kube-api-access-nkjkf\") pod \"maas-api-revoke-sub-265722de-ff6lg\" (UID: \"c5223feb-e5f0-4e64-95a9-c7f4235e99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"
Nov 02 12:17:22.302348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:22.302287    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"
Nov 02 12:17:22.684373 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:17:22.684342    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5223feb_e5f0_4e64_95a9_c7f4235e99b2.slice/crio-d6768bceec197a4281b028528feadb7eb3f8b3b20c7ecf323ab3430d620222b4 WatchSource:0}: Error finding container d6768bceec197a4281b028528feadb7eb3f8b3b20c7ecf323ab3430d620222b4: Status 404 returned error can't find the container with id d6768bceec197a4281b028528feadb7eb3f8b3b20c7ecf323ab3430d620222b4
Nov 02 12:17:22.685227 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:22.685206    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"]
Nov 02 12:17:23.073647 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:23.073609    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg" event={"ID":"c5223feb-e5f0-4e64-95a9-c7f4235e99b2","Type":"ContainerStarted","Data":"95c37f04c05d7c49258e541108dc055e189645edcfb7be65f2e9118b5804c4fc"}
Nov 02 12:17:23.073841 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:23.073689    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg" event={"ID":"c5223feb-e5f0-4e64-95a9-c7f4235e99b2","Type":"ContainerStarted","Data":"d6768bceec197a4281b028528feadb7eb3f8b3b20c7ecf323ab3430d620222b4"}
Nov 02 12:17:23.191139 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:23.191091    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg" podStartSLOduration=2.191078573 podStartE2EDuration="2.191078573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:17:23.190163584 +0000 UTC m=+1271.753872031" watchObservedRunningTime="2026-11-02 12:17:23.191078573 +0000 UTC m=+1271.754786981"
Nov 02 12:17:24.085471 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:24.085438    4464 generic.go:356] "Generic (PLEG): container finished" podID="c5223feb-e5f0-4e64-95a9-c7f4235e99b2" containerID="95c37f04c05d7c49258e541108dc055e189645edcfb7be65f2e9118b5804c4fc" exitCode=0
Nov 02 12:17:24.085856 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:24.085487    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg" event={"ID":"c5223feb-e5f0-4e64-95a9-c7f4235e99b2","Type":"ContainerDied","Data":"95c37f04c05d7c49258e541108dc055e189645edcfb7be65f2e9118b5804c4fc"}
Nov 02 12:17:25.244770 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:25.244742    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"
Nov 02 12:17:25.298404 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:25.298378    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-kube-api-access-nkjkf\" (UniqueName: \"kubernetes.io/projected/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-kube-api-access-nkjkf\") pod \"c5223feb-e5f0-4e64-95a9-c7f4235e99b2\" (UID: \"c5223feb-e5f0-4e64-95a9-c7f4235e99b2\") "
Nov 02 12:17:25.298557 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:25.298541    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-maas-api-xks-ca\") pod \"c5223feb-e5f0-4e64-95a9-c7f4235e99b2\" (UID: \"c5223feb-e5f0-4e64-95a9-c7f4235e99b2\") "
Nov 02 12:17:25.298629 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:25.298589    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-maas-api-openshift-ca\") pod \"c5223feb-e5f0-4e64-95a9-c7f4235e99b2\" (UID: \"c5223feb-e5f0-4e64-95a9-c7f4235e99b2\") "
Nov 02 12:17:25.299058 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:25.299033    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-maas-api-openshift-ca" pod "c5223feb-e5f0-4e64-95a9-c7f4235e99b2" (UID: "c5223feb-e5f0-4e64-95a9-c7f4235e99b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:17:25.300640 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:25.300605    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-maas-api-xks-ca" pod "c5223feb-e5f0-4e64-95a9-c7f4235e99b2" (UID: "c5223feb-e5f0-4e64-95a9-c7f4235e99b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:17:25.300740 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:25.300649    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-kube-api-access-nkjkf" pod "c5223feb-e5f0-4e64-95a9-c7f4235e99b2" (UID: "c5223feb-e5f0-4e64-95a9-c7f4235e99b2"). InnerVolumeSpecName "kube-api-access-nkjkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:17:25.400563 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:25.400482    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:25.400563 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:25.400523    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:25.400563 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:25.400539    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkjkf\" (UniqueName: \"kubernetes.io/projected/c5223feb-e5f0-4e64-95a9-c7f4235e99b2-kube-api-access-nkjkf\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:26.100729 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:26.100695    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"
Nov 02 12:17:26.100919 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:26.100738    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg" event={"ID":"c5223feb-e5f0-4e64-95a9-c7f4235e99b2","Type":"ContainerDied","Data":"d6768bceec197a4281b028528feadb7eb3f8b3b20c7ecf323ab3430d620222b4"}
Nov 02 12:17:26.100919 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:26.100795    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6768bceec197a4281b028528feadb7eb3f8b3b20c7ecf323ab3430d620222b4"
Nov 02 12:17:28.358499 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:17:28.358455    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:17:32.788488 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:32.788453    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"]
Nov 02 12:17:32.789145 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:32.789128    4464 cpu_manager.go:414] "containerMap: removing container" podUID="c5223feb-e5f0-4e64-95a9-c7f4235e99b2" containerName="revoke-keys"
Nov 02 12:17:32.789203 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:32.789146    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5223feb-e5f0-4e64-95a9-c7f4235e99b2" containerName="revoke-keys"
Nov 02 12:17:32.789274 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:32.789265    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5223feb-e5f0-4e64-95a9-c7f4235e99b2" containerName="revoke-keys"
Nov 02 12:17:32.792871 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:32.792853    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"
Nov 02 12:17:32.903689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:32.903660    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:17:32.903689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:32.903676    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:17:32.903689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:32.903666    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:17:32.903944 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:32.903714    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:17:32.908249 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:32.908229    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"]
Nov 02 12:17:32.913427 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:32.913401    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93497097-faf6-4ede-ab05-d5e1a398051a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95eb25be-kmttt\" (UID: \"93497097-faf6-4ede-ab05-d5e1a398051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"
Nov 02 12:17:32.913585 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:32.913569    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjcft\" (UniqueName: \"kubernetes.io/projected/93497097-faf6-4ede-ab05-d5e1a398051a-kube-api-access-rjcft\") pod \"maas-api-revoke-sub-95eb25be-kmttt\" (UID: \"93497097-faf6-4ede-ab05-d5e1a398051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"
Nov 02 12:17:32.913639 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:32.913597    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93497097-faf6-4ede-ab05-d5e1a398051a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95eb25be-kmttt\" (UID: \"93497097-faf6-4ede-ab05-d5e1a398051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"
Nov 02 12:17:33.014255 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:33.014226    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93497097-faf6-4ede-ab05-d5e1a398051a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95eb25be-kmttt\" (UID: \"93497097-faf6-4ede-ab05-d5e1a398051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"
Nov 02 12:17:33.014436 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:33.014307    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjcft\" (UniqueName: \"kubernetes.io/projected/93497097-faf6-4ede-ab05-d5e1a398051a-kube-api-access-rjcft\") pod \"maas-api-revoke-sub-95eb25be-kmttt\" (UID: \"93497097-faf6-4ede-ab05-d5e1a398051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"
Nov 02 12:17:33.014436 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:33.014329    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93497097-faf6-4ede-ab05-d5e1a398051a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95eb25be-kmttt\" (UID: \"93497097-faf6-4ede-ab05-d5e1a398051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"
Nov 02 12:17:33.014893 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:33.014872    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93497097-faf6-4ede-ab05-d5e1a398051a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95eb25be-kmttt\" (UID: \"93497097-faf6-4ede-ab05-d5e1a398051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"
Nov 02 12:17:33.016604 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:33.016584    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93497097-faf6-4ede-ab05-d5e1a398051a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95eb25be-kmttt\" (UID: \"93497097-faf6-4ede-ab05-d5e1a398051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"
Nov 02 12:17:33.076023 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:33.075949    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjcft\" (UniqueName: \"kubernetes.io/projected/93497097-faf6-4ede-ab05-d5e1a398051a-kube-api-access-rjcft\") pod \"maas-api-revoke-sub-95eb25be-kmttt\" (UID: \"93497097-faf6-4ede-ab05-d5e1a398051a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"
Nov 02 12:17:33.107191 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:33.107172    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"
Nov 02 12:17:33.487769 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:17:33.487731    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93497097_faf6_4ede_ab05_d5e1a398051a.slice/crio-0fd7e9558caaeb2e73e6629146103d4b22c86fb344453c492566ed20887474f0 WatchSource:0}: Error finding container 0fd7e9558caaeb2e73e6629146103d4b22c86fb344453c492566ed20887474f0: Status 404 returned error can't find the container with id 0fd7e9558caaeb2e73e6629146103d4b22c86fb344453c492566ed20887474f0
Nov 02 12:17:33.490198 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:33.490175    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"]
Nov 02 12:17:34.203862 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:34.203775    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt" event={"ID":"93497097-faf6-4ede-ab05-d5e1a398051a","Type":"ContainerStarted","Data":"a43f1833c93358568792d2faa5092208470f43afebb04357c26672466b470b03"}
Nov 02 12:17:34.203862 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:34.203833    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt" event={"ID":"93497097-faf6-4ede-ab05-d5e1a398051a","Type":"ContainerStarted","Data":"0fd7e9558caaeb2e73e6629146103d4b22c86fb344453c492566ed20887474f0"}
Nov 02 12:17:34.323788 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:34.323486    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt" podStartSLOduration=2.323472168 podStartE2EDuration="2.323472168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:17:34.321150202 +0000 UTC m=+1282.884858637" watchObservedRunningTime="2026-11-02 12:17:34.323472168 +0000 UTC m=+1282.887180577"
Nov 02 12:17:35.013417 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:35.013382    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"]
Nov 02 12:17:35.017642 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:35.017625    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"
Nov 02 12:17:35.075605 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:35.075576    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"]
Nov 02 12:17:35.139277 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:35.139243    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c618798-05ff-4856-809a-377c963667ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-003bee73-dphsq\" (UID: \"0c618798-05ff-4856-809a-377c963667ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"
Nov 02 12:17:35.139399 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:35.139367    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c618798-05ff-4856-809a-377c963667ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-003bee73-dphsq\" (UID: \"0c618798-05ff-4856-809a-377c963667ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"
Nov 02 12:17:35.139475 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:35.139455    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcjnc\" (UniqueName: \"kubernetes.io/projected/0c618798-05ff-4856-809a-377c963667ea-kube-api-access-fcjnc\") pod \"maas-api-revoke-sub-003bee73-dphsq\" (UID: \"0c618798-05ff-4856-809a-377c963667ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"
Nov 02 12:17:35.213418 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:35.213387    4464 generic.go:356] "Generic (PLEG): container finished" podID="93497097-faf6-4ede-ab05-d5e1a398051a" containerID="a43f1833c93358568792d2faa5092208470f43afebb04357c26672466b470b03" exitCode=0
Nov 02 12:17:35.213815 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:35.213472    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt" event={"ID":"93497097-faf6-4ede-ab05-d5e1a398051a","Type":"ContainerDied","Data":"a43f1833c93358568792d2faa5092208470f43afebb04357c26672466b470b03"}
Nov 02 12:17:35.240523 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:35.240490    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c618798-05ff-4856-809a-377c963667ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-003bee73-dphsq\" (UID: \"0c618798-05ff-4856-809a-377c963667ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"
Nov 02 12:17:35.240648 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:35.240578    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcjnc\" (UniqueName: \"kubernetes.io/projected/0c618798-05ff-4856-809a-377c963667ea-kube-api-access-fcjnc\") pod \"maas-api-revoke-sub-003bee73-dphsq\" (UID: \"0c618798-05ff-4856-809a-377c963667ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"
Nov 02 12:17:35.240727 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:35.240711    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c618798-05ff-4856-809a-377c963667ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-003bee73-dphsq\" (UID: \"0c618798-05ff-4856-809a-377c963667ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"
Nov 02 12:17:35.241292 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:35.241269    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c618798-05ff-4856-809a-377c963667ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-003bee73-dphsq\" (UID: \"0c618798-05ff-4856-809a-377c963667ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"
Nov 02 12:17:35.243011 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:35.242990    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c618798-05ff-4856-809a-377c963667ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-003bee73-dphsq\" (UID: \"0c618798-05ff-4856-809a-377c963667ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"
Nov 02 12:17:35.302922 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:35.302866    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcjnc\" (UniqueName: \"kubernetes.io/projected/0c618798-05ff-4856-809a-377c963667ea-kube-api-access-fcjnc\") pod \"maas-api-revoke-sub-003bee73-dphsq\" (UID: \"0c618798-05ff-4856-809a-377c963667ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"
Nov 02 12:17:35.331460 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:35.331437    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"
Nov 02 12:17:35.715060 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:17:35.715028    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c618798_05ff_4856_809a_377c963667ea.slice/crio-e23f84ee0e2b834f935146f74da6a007f014ec7cb435297d3a1f487dfe553378 WatchSource:0}: Error finding container e23f84ee0e2b834f935146f74da6a007f014ec7cb435297d3a1f487dfe553378: Status 404 returned error can't find the container with id e23f84ee0e2b834f935146f74da6a007f014ec7cb435297d3a1f487dfe553378
Nov 02 12:17:35.715915 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:35.715890    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"]
Nov 02 12:17:36.223163 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:36.223122    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq" event={"ID":"0c618798-05ff-4856-809a-377c963667ea","Type":"ContainerStarted","Data":"92607e1cdba75aa52826edfebee09b63f5bd28f2fdc01bacaa9bfaade2e734f7"}
Nov 02 12:17:36.223563 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:36.223224    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq" event={"ID":"0c618798-05ff-4856-809a-377c963667ea","Type":"ContainerStarted","Data":"e23f84ee0e2b834f935146f74da6a007f014ec7cb435297d3a1f487dfe553378"}
Nov 02 12:17:36.346543 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:36.346492    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq" podStartSLOduration=2.346474569 podStartE2EDuration="2.346474569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:17:36.34420418 +0000 UTC m=+1284.907912588" watchObservedRunningTime="2026-11-02 12:17:36.346474569 +0000 UTC m=+1284.910183337"
Nov 02 12:17:36.372501 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:36.372480    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"
Nov 02 12:17:36.462500 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:36.462469    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93497097-faf6-4ede-ab05-d5e1a398051a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93497097-faf6-4ede-ab05-d5e1a398051a-maas-api-openshift-ca\") pod \"93497097-faf6-4ede-ab05-d5e1a398051a\" (UID: \"93497097-faf6-4ede-ab05-d5e1a398051a\") "
Nov 02 12:17:36.462672 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:36.462601    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93497097-faf6-4ede-ab05-d5e1a398051a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93497097-faf6-4ede-ab05-d5e1a398051a-maas-api-xks-ca\") pod \"93497097-faf6-4ede-ab05-d5e1a398051a\" (UID: \"93497097-faf6-4ede-ab05-d5e1a398051a\") "
Nov 02 12:17:36.462672 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:36.462661    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93497097-faf6-4ede-ab05-d5e1a398051a-kube-api-access-rjcft\" (UniqueName: \"kubernetes.io/projected/93497097-faf6-4ede-ab05-d5e1a398051a-kube-api-access-rjcft\") pod \"93497097-faf6-4ede-ab05-d5e1a398051a\" (UID: \"93497097-faf6-4ede-ab05-d5e1a398051a\") "
Nov 02 12:17:36.462873 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:36.462851    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93497097-faf6-4ede-ab05-d5e1a398051a-maas-api-openshift-ca" pod "93497097-faf6-4ede-ab05-d5e1a398051a" (UID: "93497097-faf6-4ede-ab05-d5e1a398051a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:17:36.463170 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:36.463149    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93497097-faf6-4ede-ab05-d5e1a398051a-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:36.464940 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:36.464914    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93497097-faf6-4ede-ab05-d5e1a398051a-kube-api-access-rjcft" pod "93497097-faf6-4ede-ab05-d5e1a398051a" (UID: "93497097-faf6-4ede-ab05-d5e1a398051a"). InnerVolumeSpecName "kube-api-access-rjcft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:17:36.464940 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:36.464925    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93497097-faf6-4ede-ab05-d5e1a398051a-maas-api-xks-ca" pod "93497097-faf6-4ede-ab05-d5e1a398051a" (UID: "93497097-faf6-4ede-ab05-d5e1a398051a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:17:36.564130 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:36.564097    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93497097-faf6-4ede-ab05-d5e1a398051a-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:36.564130 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:36.564123    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjcft\" (UniqueName: \"kubernetes.io/projected/93497097-faf6-4ede-ab05-d5e1a398051a-kube-api-access-rjcft\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:37.231724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:37.231692    4464 generic.go:356] "Generic (PLEG): container finished" podID="0c618798-05ff-4856-809a-377c963667ea" containerID="92607e1cdba75aa52826edfebee09b63f5bd28f2fdc01bacaa9bfaade2e734f7" exitCode=0
Nov 02 12:17:37.232209 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:37.231773    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq" event={"ID":"0c618798-05ff-4856-809a-377c963667ea","Type":"ContainerDied","Data":"92607e1cdba75aa52826edfebee09b63f5bd28f2fdc01bacaa9bfaade2e734f7"}
Nov 02 12:17:37.233254 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:37.233236    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"
Nov 02 12:17:37.233369 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:37.233276    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt" event={"ID":"93497097-faf6-4ede-ab05-d5e1a398051a","Type":"ContainerDied","Data":"0fd7e9558caaeb2e73e6629146103d4b22c86fb344453c492566ed20887474f0"}
Nov 02 12:17:37.233369 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:37.233315    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fd7e9558caaeb2e73e6629146103d4b22c86fb344453c492566ed20887474f0"
Nov 02 12:17:38.382145 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:38.382120    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"
Nov 02 12:17:38.486233 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:38.486208    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c618798-05ff-4856-809a-377c963667ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c618798-05ff-4856-809a-377c963667ea-maas-api-openshift-ca\") pod \"0c618798-05ff-4856-809a-377c963667ea\" (UID: \"0c618798-05ff-4856-809a-377c963667ea\") "
Nov 02 12:17:38.486388 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:38.486325    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c618798-05ff-4856-809a-377c963667ea-kube-api-access-fcjnc\" (UniqueName: \"kubernetes.io/projected/0c618798-05ff-4856-809a-377c963667ea-kube-api-access-fcjnc\") pod \"0c618798-05ff-4856-809a-377c963667ea\" (UID: \"0c618798-05ff-4856-809a-377c963667ea\") "
Nov 02 12:17:38.486449 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:38.486417    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c618798-05ff-4856-809a-377c963667ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c618798-05ff-4856-809a-377c963667ea-maas-api-xks-ca\") pod \"0c618798-05ff-4856-809a-377c963667ea\" (UID: \"0c618798-05ff-4856-809a-377c963667ea\") "
Nov 02 12:17:38.486635 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:38.486612    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c618798-05ff-4856-809a-377c963667ea-maas-api-openshift-ca" pod "0c618798-05ff-4856-809a-377c963667ea" (UID: "0c618798-05ff-4856-809a-377c963667ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:17:38.486840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:38.486824    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c618798-05ff-4856-809a-377c963667ea-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:38.488335 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:38.488316    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c618798-05ff-4856-809a-377c963667ea-kube-api-access-fcjnc" pod "0c618798-05ff-4856-809a-377c963667ea" (UID: "0c618798-05ff-4856-809a-377c963667ea"). InnerVolumeSpecName "kube-api-access-fcjnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:17:38.488417 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:38.488338    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c618798-05ff-4856-809a-377c963667ea-maas-api-xks-ca" pod "0c618798-05ff-4856-809a-377c963667ea" (UID: "0c618798-05ff-4856-809a-377c963667ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:17:38.587990 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:38.587943    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c618798-05ff-4856-809a-377c963667ea-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:38.587990 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:38.587986    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcjnc\" (UniqueName: \"kubernetes.io/projected/0c618798-05ff-4856-809a-377c963667ea-kube-api-access-fcjnc\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:39.249421 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:39.249391    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"
Nov 02 12:17:39.249587 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:39.249423    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq" event={"ID":"0c618798-05ff-4856-809a-377c963667ea","Type":"ContainerDied","Data":"e23f84ee0e2b834f935146f74da6a007f014ec7cb435297d3a1f487dfe553378"}
Nov 02 12:17:39.249587 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:39.249468    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e23f84ee0e2b834f935146f74da6a007f014ec7cb435297d3a1f487dfe553378"
Nov 02 12:17:41.151085 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:41.151054    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:17:43.853294 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:43.853243    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv" podUID="d829dab9-d274-485e-bef3-a29c5ddfff22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 02 12:17:43.901586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:43.878577    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf" podUID="ff6c0679-4d25-499c-809d-4e22187c23ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded"
Nov 02 12:17:44.059999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:44.059095    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"]
Nov 02 12:17:44.085693 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:44.085655    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-481cb7b8-st2cn"]
Nov 02 12:17:45.586307 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:45.586277    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb6bf60-cf44-4fff-ac7b-0995fced611e" path="/var/lib/kubelet/pods/6cb6bf60-cf44-4fff-ac7b-0995fced611e/volumes"
Nov 02 12:17:50.187469 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.187436    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"]
Nov 02 12:17:50.188055 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.188041    4464 cpu_manager.go:414] "containerMap: removing container" podUID="0c618798-05ff-4856-809a-377c963667ea" containerName="revoke-keys"
Nov 02 12:17:50.188116 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.188055    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c618798-05ff-4856-809a-377c963667ea" containerName="revoke-keys"
Nov 02 12:17:50.188116 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.188066    4464 cpu_manager.go:414] "containerMap: removing container" podUID="93497097-faf6-4ede-ab05-d5e1a398051a" containerName="revoke-keys"
Nov 02 12:17:50.188116 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.188071    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93497097-faf6-4ede-ab05-d5e1a398051a" containerName="revoke-keys"
Nov 02 12:17:50.188211 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.188187    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c618798-05ff-4856-809a-377c963667ea" containerName="revoke-keys"
Nov 02 12:17:50.188211 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.188195    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="93497097-faf6-4ede-ab05-d5e1a398051a" containerName="revoke-keys"
Nov 02 12:17:50.193130 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.193114    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"
Nov 02 12:17:50.250182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.250156    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"]
Nov 02 12:17:50.295630 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.295596    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:17:50.295787 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.295635    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:17:50.295834 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.295779    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:17:50.295834 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.295806    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:17:50.296190 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.296170    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b67762-1dbe-464d-b137-d0392e00781d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-306303de-cvn92\" (UID: \"65b67762-1dbe-464d-b137-d0392e00781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"
Nov 02 12:17:50.296264 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.296248    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b67762-1dbe-464d-b137-d0392e00781d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-306303de-cvn92\" (UID: \"65b67762-1dbe-464d-b137-d0392e00781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"
Nov 02 12:17:50.296405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.296386    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s822\" (UniqueName: \"kubernetes.io/projected/65b67762-1dbe-464d-b137-d0392e00781d-kube-api-access-4s822\") pod \"maas-api-revoke-sub-306303de-cvn92\" (UID: \"65b67762-1dbe-464d-b137-d0392e00781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"
Nov 02 12:17:50.397279 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.397240    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s822\" (UniqueName: \"kubernetes.io/projected/65b67762-1dbe-464d-b137-d0392e00781d-kube-api-access-4s822\") pod \"maas-api-revoke-sub-306303de-cvn92\" (UID: \"65b67762-1dbe-464d-b137-d0392e00781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"
Nov 02 12:17:50.397452 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.397342    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b67762-1dbe-464d-b137-d0392e00781d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-306303de-cvn92\" (UID: \"65b67762-1dbe-464d-b137-d0392e00781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"
Nov 02 12:17:50.397452 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.397417    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b67762-1dbe-464d-b137-d0392e00781d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-306303de-cvn92\" (UID: \"65b67762-1dbe-464d-b137-d0392e00781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"
Nov 02 12:17:50.397957 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.397931    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b67762-1dbe-464d-b137-d0392e00781d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-306303de-cvn92\" (UID: \"65b67762-1dbe-464d-b137-d0392e00781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"
Nov 02 12:17:50.399688 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.399664    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b67762-1dbe-464d-b137-d0392e00781d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-306303de-cvn92\" (UID: \"65b67762-1dbe-464d-b137-d0392e00781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"
Nov 02 12:17:50.459663 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.459604    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s822\" (UniqueName: \"kubernetes.io/projected/65b67762-1dbe-464d-b137-d0392e00781d-kube-api-access-4s822\") pod \"maas-api-revoke-sub-306303de-cvn92\" (UID: \"65b67762-1dbe-464d-b137-d0392e00781d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"
Nov 02 12:17:50.506930 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.506907    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"
Nov 02 12:17:50.884405 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:17:50.884364    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b67762_1dbe_464d_b137_d0392e00781d.slice/crio-fff4a5a8af0211f7182a56075b96fd29df4c1fa6e4eae2f3b9f35da717bc944c WatchSource:0}: Error finding container fff4a5a8af0211f7182a56075b96fd29df4c1fa6e4eae2f3b9f35da717bc944c: Status 404 returned error can't find the container with id fff4a5a8af0211f7182a56075b96fd29df4c1fa6e4eae2f3b9f35da717bc944c
Nov 02 12:17:50.886741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:50.886715    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"]
Nov 02 12:17:51.279340 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:51.279307    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"]
Nov 02 12:17:51.489033 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:51.488993    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92" event={"ID":"65b67762-1dbe-464d-b137-d0392e00781d","Type":"ContainerStarted","Data":"808490c5d1a889b13ac34c8031a48d67aa924bd663ca98dd461721c63d1badb6"}
Nov 02 12:17:51.489239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:51.489071    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92" event={"ID":"65b67762-1dbe-464d-b137-d0392e00781d","Type":"ContainerStarted","Data":"fff4a5a8af0211f7182a56075b96fd29df4c1fa6e4eae2f3b9f35da717bc944c"}
Nov 02 12:17:51.606250 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:51.606135    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92" podStartSLOduration=1.6061179110000001 podStartE2EDuration="1.606117911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:17:51.60596757 +0000 UTC m=+1300.169675981" watchObservedRunningTime="2026-11-02 12:17:51.606117911 +0000 UTC m=+1300.169826322"
Nov 02 12:17:52.049250 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:52.049217    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"]
Nov 02 12:17:52.053638 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:52.053611    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86dbb9aa-6gczb"]
Nov 02 12:17:52.296655 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:52.296628    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"]
Nov 02 12:17:52.301060 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:52.301003    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"
Nov 02 12:17:52.359348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:52.359318    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"]
Nov 02 12:17:52.421872 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:52.421843    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc135360-afd8-4b66-97fd-92d9f60c8611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba1c4212-qrcpr\" (UID: \"fc135360-afd8-4b66-97fd-92d9f60c8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"
Nov 02 12:17:52.422055 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:52.421890    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv7sj\" (UniqueName: \"kubernetes.io/projected/fc135360-afd8-4b66-97fd-92d9f60c8611-kube-api-access-zv7sj\") pod \"maas-api-revoke-sub-ba1c4212-qrcpr\" (UID: \"fc135360-afd8-4b66-97fd-92d9f60c8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"
Nov 02 12:17:52.422142 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:52.422073    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc135360-afd8-4b66-97fd-92d9f60c8611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba1c4212-qrcpr\" (UID: \"fc135360-afd8-4b66-97fd-92d9f60c8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"
Nov 02 12:17:52.497566 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:52.497538    4464 generic.go:356] "Generic (PLEG): container finished" podID="65b67762-1dbe-464d-b137-d0392e00781d" containerID="808490c5d1a889b13ac34c8031a48d67aa924bd663ca98dd461721c63d1badb6" exitCode=0
Nov 02 12:17:52.497697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:52.497618    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92" event={"ID":"65b67762-1dbe-464d-b137-d0392e00781d","Type":"ContainerDied","Data":"808490c5d1a889b13ac34c8031a48d67aa924bd663ca98dd461721c63d1badb6"}
Nov 02 12:17:52.523029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:52.523004    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc135360-afd8-4b66-97fd-92d9f60c8611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba1c4212-qrcpr\" (UID: \"fc135360-afd8-4b66-97fd-92d9f60c8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"
Nov 02 12:17:52.523161 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:52.523146    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc135360-afd8-4b66-97fd-92d9f60c8611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba1c4212-qrcpr\" (UID: \"fc135360-afd8-4b66-97fd-92d9f60c8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"
Nov 02 12:17:52.523212 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:52.523176    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv7sj\" (UniqueName: \"kubernetes.io/projected/fc135360-afd8-4b66-97fd-92d9f60c8611-kube-api-access-zv7sj\") pod \"maas-api-revoke-sub-ba1c4212-qrcpr\" (UID: \"fc135360-afd8-4b66-97fd-92d9f60c8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"
Nov 02 12:17:52.523602 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:52.523581    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc135360-afd8-4b66-97fd-92d9f60c8611-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba1c4212-qrcpr\" (UID: \"fc135360-afd8-4b66-97fd-92d9f60c8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"
Nov 02 12:17:52.525416 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:52.525398    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc135360-afd8-4b66-97fd-92d9f60c8611-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba1c4212-qrcpr\" (UID: \"fc135360-afd8-4b66-97fd-92d9f60c8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"
Nov 02 12:17:52.585142 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:52.585076    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv7sj\" (UniqueName: \"kubernetes.io/projected/fc135360-afd8-4b66-97fd-92d9f60c8611-kube-api-access-zv7sj\") pod \"maas-api-revoke-sub-ba1c4212-qrcpr\" (UID: \"fc135360-afd8-4b66-97fd-92d9f60c8611\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"
Nov 02 12:17:52.614465 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:52.614444    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"
Nov 02 12:17:53.001265 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:17:53.001233    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc135360_afd8_4b66_97fd_92d9f60c8611.slice/crio-86730b0d47de3a94bdca02a4dab9bce7c225fd1a6e98a78e602cd2b997952884 WatchSource:0}: Error finding container 86730b0d47de3a94bdca02a4dab9bce7c225fd1a6e98a78e602cd2b997952884: Status 404 returned error can't find the container with id 86730b0d47de3a94bdca02a4dab9bce7c225fd1a6e98a78e602cd2b997952884
Nov 02 12:17:53.002438 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:53.002417    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"]
Nov 02 12:17:53.606956 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:53.606147    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baa262bd-0aa2-4022-be8d-4c0efc3c4f57" path="/var/lib/kubelet/pods/baa262bd-0aa2-4022-be8d-4c0efc3c4f57/volumes"
Nov 02 12:17:53.606956 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:53.606667    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr" event={"ID":"fc135360-afd8-4b66-97fd-92d9f60c8611","Type":"ContainerStarted","Data":"2b75253483ba9bf714c57c4d6bcdfb319ddae7d739bb20db4fa95223582d2882"}
Nov 02 12:17:53.606956 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:53.606730    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr" event={"ID":"fc135360-afd8-4b66-97fd-92d9f60c8611","Type":"ContainerStarted","Data":"86730b0d47de3a94bdca02a4dab9bce7c225fd1a6e98a78e602cd2b997952884"}
Nov 02 12:17:53.704768 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:53.704708    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr" podStartSLOduration=1.70468871 podStartE2EDuration="1.70468871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:17:53.70163523 +0000 UTC m=+1302.265343640" watchObservedRunningTime="2026-11-02 12:17:53.70468871 +0000 UTC m=+1302.268397120"
Nov 02 12:17:53.788122 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:53.788092    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"
Nov 02 12:17:53.861289 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:53.861205    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65b67762-1dbe-464d-b137-d0392e00781d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b67762-1dbe-464d-b137-d0392e00781d-maas-api-xks-ca\") pod \"65b67762-1dbe-464d-b137-d0392e00781d\" (UID: \"65b67762-1dbe-464d-b137-d0392e00781d\") "
Nov 02 12:17:53.861289 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:53.861271    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65b67762-1dbe-464d-b137-d0392e00781d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b67762-1dbe-464d-b137-d0392e00781d-maas-api-openshift-ca\") pod \"65b67762-1dbe-464d-b137-d0392e00781d\" (UID: \"65b67762-1dbe-464d-b137-d0392e00781d\") "
Nov 02 12:17:53.861516 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:53.861481    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b67762-1dbe-464d-b137-d0392e00781d-kube-api-access-4s822\" (UniqueName: \"kubernetes.io/projected/65b67762-1dbe-464d-b137-d0392e00781d-kube-api-access-4s822\") pod \"65b67762-1dbe-464d-b137-d0392e00781d\" (UID: \"65b67762-1dbe-464d-b137-d0392e00781d\") "
Nov 02 12:17:53.861702 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:53.861671    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b67762-1dbe-464d-b137-d0392e00781d-maas-api-openshift-ca" pod "65b67762-1dbe-464d-b137-d0392e00781d" (UID: "65b67762-1dbe-464d-b137-d0392e00781d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:17:53.862077 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:53.862049    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b67762-1dbe-464d-b137-d0392e00781d-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:53.863587 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:53.863558    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65b67762-1dbe-464d-b137-d0392e00781d-maas-api-xks-ca" pod "65b67762-1dbe-464d-b137-d0392e00781d" (UID: "65b67762-1dbe-464d-b137-d0392e00781d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:17:53.864097 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:53.864072    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b67762-1dbe-464d-b137-d0392e00781d-kube-api-access-4s822" pod "65b67762-1dbe-464d-b137-d0392e00781d" (UID: "65b67762-1dbe-464d-b137-d0392e00781d"). InnerVolumeSpecName "kube-api-access-4s822". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:17:53.965566 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:53.965542    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s822\" (UniqueName: \"kubernetes.io/projected/65b67762-1dbe-464d-b137-d0392e00781d-kube-api-access-4s822\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:53.965656 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:53.965574    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b67762-1dbe-464d-b137-d0392e00781d-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:54.639098 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:54.632448    4464 generic.go:356] "Generic (PLEG): container finished" podID="fc135360-afd8-4b66-97fd-92d9f60c8611" containerID="2b75253483ba9bf714c57c4d6bcdfb319ddae7d739bb20db4fa95223582d2882" exitCode=0
Nov 02 12:17:54.639098 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:54.632902    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr" event={"ID":"fc135360-afd8-4b66-97fd-92d9f60c8611","Type":"ContainerDied","Data":"2b75253483ba9bf714c57c4d6bcdfb319ddae7d739bb20db4fa95223582d2882"}
Nov 02 12:17:54.646003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:54.642666    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92" event={"ID":"65b67762-1dbe-464d-b137-d0392e00781d","Type":"ContainerDied","Data":"fff4a5a8af0211f7182a56075b96fd29df4c1fa6e4eae2f3b9f35da717bc944c"}
Nov 02 12:17:54.646003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:54.642721    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fff4a5a8af0211f7182a56075b96fd29df4c1fa6e4eae2f3b9f35da717bc944c"
Nov 02 12:17:54.646003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:54.642785    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"
Nov 02 12:17:56.667773 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:56.667743    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"
Nov 02 12:17:56.703916 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:56.703877    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr" event={"ID":"fc135360-afd8-4b66-97fd-92d9f60c8611","Type":"ContainerDied","Data":"86730b0d47de3a94bdca02a4dab9bce7c225fd1a6e98a78e602cd2b997952884"}
Nov 02 12:17:56.704176 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:56.704157    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86730b0d47de3a94bdca02a4dab9bce7c225fd1a6e98a78e602cd2b997952884"
Nov 02 12:17:56.704342 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:56.704329    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"
Nov 02 12:17:56.802356 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:56.802323    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc135360-afd8-4b66-97fd-92d9f60c8611-kube-api-access-zv7sj\" (UniqueName: \"kubernetes.io/projected/fc135360-afd8-4b66-97fd-92d9f60c8611-kube-api-access-zv7sj\") pod \"fc135360-afd8-4b66-97fd-92d9f60c8611\" (UID: \"fc135360-afd8-4b66-97fd-92d9f60c8611\") "
Nov 02 12:17:56.802704 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:56.802682    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc135360-afd8-4b66-97fd-92d9f60c8611-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc135360-afd8-4b66-97fd-92d9f60c8611-maas-api-openshift-ca\") pod \"fc135360-afd8-4b66-97fd-92d9f60c8611\" (UID: \"fc135360-afd8-4b66-97fd-92d9f60c8611\") "
Nov 02 12:17:56.803384 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:56.803360    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc135360-afd8-4b66-97fd-92d9f60c8611-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc135360-afd8-4b66-97fd-92d9f60c8611-maas-api-xks-ca\") pod \"fc135360-afd8-4b66-97fd-92d9f60c8611\" (UID: \"fc135360-afd8-4b66-97fd-92d9f60c8611\") "
Nov 02 12:17:56.803763 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:56.802998    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc135360-afd8-4b66-97fd-92d9f60c8611-maas-api-openshift-ca" pod "fc135360-afd8-4b66-97fd-92d9f60c8611" (UID: "fc135360-afd8-4b66-97fd-92d9f60c8611"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:17:56.804135 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:56.804112    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc135360-afd8-4b66-97fd-92d9f60c8611-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:56.805591 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:56.805564    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc135360-afd8-4b66-97fd-92d9f60c8611-kube-api-access-zv7sj" pod "fc135360-afd8-4b66-97fd-92d9f60c8611" (UID: "fc135360-afd8-4b66-97fd-92d9f60c8611"). InnerVolumeSpecName "kube-api-access-zv7sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:17:56.805712 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:56.805691    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc135360-afd8-4b66-97fd-92d9f60c8611-maas-api-xks-ca" pod "fc135360-afd8-4b66-97fd-92d9f60c8611" (UID: "fc135360-afd8-4b66-97fd-92d9f60c8611"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:17:56.905179 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:56.905099    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc135360-afd8-4b66-97fd-92d9f60c8611-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:56.905179 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:56.905128    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv7sj\" (UniqueName: \"kubernetes.io/projected/fc135360-afd8-4b66-97fd-92d9f60c8611-kube-api-access-zv7sj\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:17:58.311600 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:17:58.311564    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:17:59.073019 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:59.059957    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"]
Nov 02 12:17:59.130056 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:59.130023    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-994ce40c-bbx6h"]
Nov 02 12:17:59.592576 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:17:59.592544    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c016e94-ec5b-499c-97af-e46396706a71" path="/var/lib/kubelet/pods/4c016e94-ec5b-499c-97af-e46396706a71/volumes"
Nov 02 12:18:00.976583 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:00.976541    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"]
Nov 02 12:18:00.977251 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:00.977233    4464 cpu_manager.go:414] "containerMap: removing container" podUID="fc135360-afd8-4b66-97fd-92d9f60c8611" containerName="revoke-keys"
Nov 02 12:18:00.977251 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:00.977250    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc135360-afd8-4b66-97fd-92d9f60c8611" containerName="revoke-keys"
Nov 02 12:18:00.977374 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:00.977266    4464 cpu_manager.go:414] "containerMap: removing container" podUID="65b67762-1dbe-464d-b137-d0392e00781d" containerName="revoke-keys"
Nov 02 12:18:00.977374 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:00.977271    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b67762-1dbe-464d-b137-d0392e00781d" containerName="revoke-keys"
Nov 02 12:18:00.977374 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:00.977368    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc135360-afd8-4b66-97fd-92d9f60c8611" containerName="revoke-keys"
Nov 02 12:18:00.977503 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:00.977377    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b67762-1dbe-464d-b137-d0392e00781d" containerName="revoke-keys"
Nov 02 12:18:00.981057 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:00.981038    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"
Nov 02 12:18:01.106996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:01.105272    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:18:01.106996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:01.105365    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:18:01.106996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:01.105445    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:18:01.106996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:01.105537    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:18:01.151368 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:01.151334    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"]
Nov 02 12:18:01.208213 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:01.208179    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db0ecee-4d9e-4639-9967-bb61236d29a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6672e2a0-nwwjx\" (UID: \"7db0ecee-4d9e-4639-9967-bb61236d29a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"
Nov 02 12:18:01.208396 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:01.208365    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt8cj\" (UniqueName: \"kubernetes.io/projected/7db0ecee-4d9e-4639-9967-bb61236d29a4-kube-api-access-nt8cj\") pod \"maas-api-revoke-sub-6672e2a0-nwwjx\" (UID: \"7db0ecee-4d9e-4639-9967-bb61236d29a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"
Nov 02 12:18:01.208455 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:01.208424    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db0ecee-4d9e-4639-9967-bb61236d29a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6672e2a0-nwwjx\" (UID: \"7db0ecee-4d9e-4639-9967-bb61236d29a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"
Nov 02 12:18:01.310689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:01.310609    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt8cj\" (UniqueName: \"kubernetes.io/projected/7db0ecee-4d9e-4639-9967-bb61236d29a4-kube-api-access-nt8cj\") pod \"maas-api-revoke-sub-6672e2a0-nwwjx\" (UID: \"7db0ecee-4d9e-4639-9967-bb61236d29a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"
Nov 02 12:18:01.310689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:01.310671    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db0ecee-4d9e-4639-9967-bb61236d29a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6672e2a0-nwwjx\" (UID: \"7db0ecee-4d9e-4639-9967-bb61236d29a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"
Nov 02 12:18:01.310877 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:01.310831    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db0ecee-4d9e-4639-9967-bb61236d29a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6672e2a0-nwwjx\" (UID: \"7db0ecee-4d9e-4639-9967-bb61236d29a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"
Nov 02 12:18:01.311505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:01.311481    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db0ecee-4d9e-4639-9967-bb61236d29a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6672e2a0-nwwjx\" (UID: \"7db0ecee-4d9e-4639-9967-bb61236d29a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"
Nov 02 12:18:01.313217 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:01.313189    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db0ecee-4d9e-4639-9967-bb61236d29a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6672e2a0-nwwjx\" (UID: \"7db0ecee-4d9e-4639-9967-bb61236d29a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"
Nov 02 12:18:01.375158 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:01.375118    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt8cj\" (UniqueName: \"kubernetes.io/projected/7db0ecee-4d9e-4639-9967-bb61236d29a4-kube-api-access-nt8cj\") pod \"maas-api-revoke-sub-6672e2a0-nwwjx\" (UID: \"7db0ecee-4d9e-4639-9967-bb61236d29a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"
Nov 02 12:18:01.618102 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:01.618026    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"
Nov 02 12:18:02.004655 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:02.004625    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"]
Nov 02 12:18:02.005139 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:18:02.005108    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db0ecee_4d9e_4639_9967_bb61236d29a4.slice/crio-4a7588ea6a9e90503cf1e11fc9ca0b4be7a36d804b8e2df20a5ba23601ae50c1 WatchSource:0}: Error finding container 4a7588ea6a9e90503cf1e11fc9ca0b4be7a36d804b8e2df20a5ba23601ae50c1: Status 404 returned error can't find the container with id 4a7588ea6a9e90503cf1e11fc9ca0b4be7a36d804b8e2df20a5ba23601ae50c1
Nov 02 12:18:02.783272 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:02.783239    4464 generic.go:356] "Generic (PLEG): container finished" podID="7db0ecee-4d9e-4639-9967-bb61236d29a4" containerID="1d9a61cc0e99188e4f1e36a0df29182ef30bf1331cfa3bc50b6b5e27dbb24952" exitCode=0
Nov 02 12:18:02.783374 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:02.783311    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx" event={"ID":"7db0ecee-4d9e-4639-9967-bb61236d29a4","Type":"ContainerDied","Data":"1d9a61cc0e99188e4f1e36a0df29182ef30bf1331cfa3bc50b6b5e27dbb24952"}
Nov 02 12:18:02.783374 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:02.783361    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx" event={"ID":"7db0ecee-4d9e-4639-9967-bb61236d29a4","Type":"ContainerStarted","Data":"4a7588ea6a9e90503cf1e11fc9ca0b4be7a36d804b8e2df20a5ba23601ae50c1"}
Nov 02 12:18:03.792419 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:03.792343    4464 generic.go:356] "Generic (PLEG): container finished" podID="86b728ba-2cb4-4e99-88b3-d6f0afd94f47" containerID="f7b4fc003c3691b7cd85a3f2dd67b9a37e88606e966e6fec8798cbf4e8023b35" exitCode=0
Nov 02 12:18:03.792906 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:03.792423    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m" event={"ID":"86b728ba-2cb4-4e99-88b3-d6f0afd94f47","Type":"ContainerDied","Data":"f7b4fc003c3691b7cd85a3f2dd67b9a37e88606e966e6fec8798cbf4e8023b35"}
Nov 02 12:18:03.793331 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:03.792951    4464 scope.go:122] "RemoveContainer" containerID="f7b4fc003c3691b7cd85a3f2dd67b9a37e88606e966e6fec8798cbf4e8023b35"
Nov 02 12:18:03.795034 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:03.795005    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:18:03.795135 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:03.795097    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:18:03.795183 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:03.795151    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:18:03.957182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:03.957157    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"
Nov 02 12:18:04.051138 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:04.051099    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db0ecee-4d9e-4639-9967-bb61236d29a4-kube-api-access-nt8cj\" (UniqueName: \"kubernetes.io/projected/7db0ecee-4d9e-4639-9967-bb61236d29a4-kube-api-access-nt8cj\") pod \"7db0ecee-4d9e-4639-9967-bb61236d29a4\" (UID: \"7db0ecee-4d9e-4639-9967-bb61236d29a4\") "
Nov 02 12:18:04.051324 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:04.051260    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7db0ecee-4d9e-4639-9967-bb61236d29a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db0ecee-4d9e-4639-9967-bb61236d29a4-maas-api-openshift-ca\") pod \"7db0ecee-4d9e-4639-9967-bb61236d29a4\" (UID: \"7db0ecee-4d9e-4639-9967-bb61236d29a4\") "
Nov 02 12:18:04.051447 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:04.051427    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7db0ecee-4d9e-4639-9967-bb61236d29a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db0ecee-4d9e-4639-9967-bb61236d29a4-maas-api-xks-ca\") pod \"7db0ecee-4d9e-4639-9967-bb61236d29a4\" (UID: \"7db0ecee-4d9e-4639-9967-bb61236d29a4\") "
Nov 02 12:18:04.051608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:04.051576    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db0ecee-4d9e-4639-9967-bb61236d29a4-maas-api-openshift-ca" pod "7db0ecee-4d9e-4639-9967-bb61236d29a4" (UID: "7db0ecee-4d9e-4639-9967-bb61236d29a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:18:04.051931 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:04.051910    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db0ecee-4d9e-4639-9967-bb61236d29a4-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:04.053295 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:04.053273    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db0ecee-4d9e-4639-9967-bb61236d29a4-kube-api-access-nt8cj" pod "7db0ecee-4d9e-4639-9967-bb61236d29a4" (UID: "7db0ecee-4d9e-4639-9967-bb61236d29a4"). InnerVolumeSpecName "kube-api-access-nt8cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:18:04.053379 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:04.053280    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7db0ecee-4d9e-4639-9967-bb61236d29a4-maas-api-xks-ca" pod "7db0ecee-4d9e-4639-9967-bb61236d29a4" (UID: "7db0ecee-4d9e-4639-9967-bb61236d29a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:18:04.153560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:04.153518    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db0ecee-4d9e-4639-9967-bb61236d29a4-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:04.153560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:04.153558    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt8cj\" (UniqueName: \"kubernetes.io/projected/7db0ecee-4d9e-4639-9967-bb61236d29a4-kube-api-access-nt8cj\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:04.802348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:04.802312    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m" event={"ID":"86b728ba-2cb4-4e99-88b3-d6f0afd94f47","Type":"ContainerStarted","Data":"7f2a9979275bacef5dbf0e995bc2d32cafc993f294bcc5bfd0777be0957aecee"}
Nov 02 12:18:04.803782 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:04.803754    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"
Nov 02 12:18:04.803905 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:04.803781    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx" event={"ID":"7db0ecee-4d9e-4639-9967-bb61236d29a4","Type":"ContainerDied","Data":"4a7588ea6a9e90503cf1e11fc9ca0b4be7a36d804b8e2df20a5ba23601ae50c1"}
Nov 02 12:18:04.803905 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:04.803826    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a7588ea6a9e90503cf1e11fc9ca0b4be7a36d804b8e2df20a5ba23601ae50c1"
Nov 02 12:18:04.985028 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:04.981085    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:18:04.985028 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:04.984255    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:18:04.985679 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:04.985648    4464 patch_prober.go:29] interesting pod/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 02 12:18:04.985799 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:04.985713    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m" podUID="86b728ba-2cb4-4e99-88b3-d6f0afd94f47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 02 12:18:05.988003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:05.985355    4464 patch_prober.go:29] interesting pod/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 02 12:18:05.988003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:05.985505    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m" podUID="86b728ba-2cb4-4e99-88b3-d6f0afd94f47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 02 12:18:06.300058 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.299943    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf" podUID="ff6c0679-4d25-499c-809d-4e22187c23ac" containerName="main" containerID="cri-o://2d2b3f52c904fa0cee2ec422c89a98fd092c138e77e0ca90d2504143ce9bdbdf" gracePeriod=45
Nov 02 12:18:06.622646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.622621    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:18:06.721853 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.721823    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-model-cache\") pod \"ff6c0679-4d25-499c-809d-4e22187c23ac\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") "
Nov 02 12:18:06.722035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.721870    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff6c0679-4d25-499c-809d-4e22187c23ac-kube-api-access-vkpps\" (UniqueName: \"kubernetes.io/projected/ff6c0679-4d25-499c-809d-4e22187c23ac-kube-api-access-vkpps\") pod \"ff6c0679-4d25-499c-809d-4e22187c23ac\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") "
Nov 02 12:18:06.722035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.721912    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff6c0679-4d25-499c-809d-4e22187c23ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff6c0679-4d25-499c-809d-4e22187c23ac-tls-certs\") pod \"ff6c0679-4d25-499c-809d-4e22187c23ac\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") "
Nov 02 12:18:06.722140 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.722038    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-dshm\") pod \"ff6c0679-4d25-499c-809d-4e22187c23ac\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") "
Nov 02 12:18:06.722140 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.722081    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-tmp-dir\") pod \"ff6c0679-4d25-499c-809d-4e22187c23ac\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") "
Nov 02 12:18:06.722140 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.722099    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-model-cache" pod "ff6c0679-4d25-499c-809d-4e22187c23ac" (UID: "ff6c0679-4d25-499c-809d-4e22187c23ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:18:06.722140 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.722116    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-home\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-home\") pod \"ff6c0679-4d25-499c-809d-4e22187c23ac\" (UID: \"ff6c0679-4d25-499c-809d-4e22187c23ac\") "
Nov 02 12:18:06.722349 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.722312    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-home" pod "ff6c0679-4d25-499c-809d-4e22187c23ac" (UID: "ff6c0679-4d25-499c-809d-4e22187c23ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:18:06.722408 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.722355    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-tmp-dir" pod "ff6c0679-4d25-499c-809d-4e22187c23ac" (UID: "ff6c0679-4d25-499c-809d-4e22187c23ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:18:06.722682 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.722663    4464 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-tmp-dir\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:06.722746 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.722684    4464 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-home\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:06.722746 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.722694    4464 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-model-cache\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:06.724178 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.724150    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6c0679-4d25-499c-809d-4e22187c23ac-kube-api-access-vkpps" pod "ff6c0679-4d25-499c-809d-4e22187c23ac" (UID: "ff6c0679-4d25-499c-809d-4e22187c23ac"). InnerVolumeSpecName "kube-api-access-vkpps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:18:06.724524 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.724500    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-dshm" pod "ff6c0679-4d25-499c-809d-4e22187c23ac" (UID: "ff6c0679-4d25-499c-809d-4e22187c23ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:18:06.724524 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.724499    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff6c0679-4d25-499c-809d-4e22187c23ac-tls-certs" pod "ff6c0679-4d25-499c-809d-4e22187c23ac" (UID: "ff6c0679-4d25-499c-809d-4e22187c23ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:18:06.820492 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.820461    4464 generic.go:356] "Generic (PLEG): container finished" podID="ff6c0679-4d25-499c-809d-4e22187c23ac" containerID="2d2b3f52c904fa0cee2ec422c89a98fd092c138e77e0ca90d2504143ce9bdbdf" exitCode=0
Nov 02 12:18:06.820633 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.820503    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"
Nov 02 12:18:06.820633 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.820543    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf" event={"ID":"ff6c0679-4d25-499c-809d-4e22187c23ac","Type":"ContainerDied","Data":"2d2b3f52c904fa0cee2ec422c89a98fd092c138e77e0ca90d2504143ce9bdbdf"}
Nov 02 12:18:06.820633 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.820585    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf" event={"ID":"ff6c0679-4d25-499c-809d-4e22187c23ac","Type":"ContainerDied","Data":"005dbf177be9a39d9179caad3b21d96456acf3d90f58c695f636f9bf0b75aa48"}
Nov 02 12:18:06.820633 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.820604    4464 scope.go:122] "RemoveContainer" containerID="2d2b3f52c904fa0cee2ec422c89a98fd092c138e77e0ca90d2504143ce9bdbdf"
Nov 02 12:18:06.824558 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.824527    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkpps\" (UniqueName: \"kubernetes.io/projected/ff6c0679-4d25-499c-809d-4e22187c23ac-kube-api-access-vkpps\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:06.824688 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.824562    4464 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff6c0679-4d25-499c-809d-4e22187c23ac-tls-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:06.824688 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.824579    4464 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff6c0679-4d25-499c-809d-4e22187c23ac-dshm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:06.831970 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.831952    4464 scope.go:122] "RemoveContainer" containerID="2d2b3f52c904fa0cee2ec422c89a98fd092c138e77e0ca90d2504143ce9bdbdf"
Nov 02 12:18:06.832281 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:18:06.832263    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2b3f52c904fa0cee2ec422c89a98fd092c138e77e0ca90d2504143ce9bdbdf\": container with ID starting with 2d2b3f52c904fa0cee2ec422c89a98fd092c138e77e0ca90d2504143ce9bdbdf not found: ID does not exist" containerID="2d2b3f52c904fa0cee2ec422c89a98fd092c138e77e0ca90d2504143ce9bdbdf"
Nov 02 12:18:06.832379 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.832288    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2b3f52c904fa0cee2ec422c89a98fd092c138e77e0ca90d2504143ce9bdbdf"} err="failed to get container status \"2d2b3f52c904fa0cee2ec422c89a98fd092c138e77e0ca90d2504143ce9bdbdf\": rpc error: code = NotFound desc = could not find container \"2d2b3f52c904fa0cee2ec422c89a98fd092c138e77e0ca90d2504143ce9bdbdf\": container with ID starting with 2d2b3f52c904fa0cee2ec422c89a98fd092c138e77e0ca90d2504143ce9bdbdf not found: ID does not exist"
Nov 02 12:18:06.984343 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.984311    4464 patch_prober.go:29] interesting pod/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 02 12:18:06.984504 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.984371    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m" podUID="86b728ba-2cb4-4e99-88b3-d6f0afd94f47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 02 12:18:06.991905 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.991870    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"]
Nov 02 12:18:06.996845 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:06.996825    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7bd72e/test-model-7bd72e-kserve-878dc8cbf-882jf"]
Nov 02 12:18:07.591046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:07.591015    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff6c0679-4d25-499c-809d-4e22187c23ac" path="/var/lib/kubelet/pods/ff6c0679-4d25-499c-809d-4e22187c23ac/volumes"
Nov 02 12:18:07.985828 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:07.985793    4464 patch_prober.go:29] interesting pod/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 02 12:18:07.986137 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:07.986109    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m" podUID="86b728ba-2cb4-4e99-88b3-d6f0afd94f47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 02 12:18:08.984341 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:08.984303    4464 patch_prober.go:29] interesting pod/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 02 12:18:08.984752 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:08.984360    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m" podUID="86b728ba-2cb4-4e99-88b3-d6f0afd94f47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 02 12:18:09.687624 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:09.687585    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:18:10.986861 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:10.986802    4464 patch_prober.go:29] interesting pod/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:18:10.988720 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:10.988684    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m" podUID="86b728ba-2cb4-4e99-88b3-d6f0afd94f47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:18:11.225337 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:11.225305    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:18:11.235295 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:11.235230    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 12:18:11.846949 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:11.846914    4464 scope.go:122] "RemoveContainer" containerID="5aaefbc70d8f79b91842fb5307d82fd4210ccf47a8f25388c9382ebe0a57ab2a"
Nov 02 12:18:11.857348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:11.857330    4464 scope.go:122] "RemoveContainer" containerID="2984ee6c41dc6c6a27157cc6888c7b66bbf65909bacdb817262ac699cbad9494"
Nov 02 12:18:11.938397 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:11.938365    4464 scope.go:122] "RemoveContainer" containerID="9cc6de1e88d1935edf60e35161df8aab8936113c54b603369d22a981d21e801f"
Nov 02 12:18:16.049918 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:16.049879    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"]
Nov 02 12:18:16.100023 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:16.099994    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d8cc74d-4fxvj"]
Nov 02 12:18:17.582594 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:17.582554    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="287bc5af-fcfd-4d80-ae6a-2325c63c0596" path="/var/lib/kubelet/pods/287bc5af-fcfd-4d80-ae6a-2325c63c0596/volumes"
Nov 02 12:18:19.708697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.708661    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"]
Nov 02 12:18:19.709703 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.709682    4464 cpu_manager.go:414] "containerMap: removing container" podUID="ff6c0679-4d25-499c-809d-4e22187c23ac" containerName="main"
Nov 02 12:18:19.709748 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.709707    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff6c0679-4d25-499c-809d-4e22187c23ac" containerName="main"
Nov 02 12:18:19.709783 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.709749    4464 cpu_manager.go:414] "containerMap: removing container" podUID="7db0ecee-4d9e-4639-9967-bb61236d29a4" containerName="revoke-keys"
Nov 02 12:18:19.709783 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.709759    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7db0ecee-4d9e-4639-9967-bb61236d29a4" containerName="revoke-keys"
Nov 02 12:18:19.709919 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.709905    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff6c0679-4d25-499c-809d-4e22187c23ac" containerName="main"
Nov 02 12:18:19.709958 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.709929    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="7db0ecee-4d9e-4639-9967-bb61236d29a4" containerName="revoke-keys"
Nov 02 12:18:19.714553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.714535    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"
Nov 02 12:18:19.768340 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.768311    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"]
Nov 02 12:18:19.813371 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.813342    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:18:19.813371 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.813356    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:18:19.813574 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.813416    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:18:19.813574 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.813435    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:18:19.836363 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.836335    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm78x\" (UniqueName: \"kubernetes.io/projected/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-kube-api-access-fm78x\") pod \"maas-api-revoke-sub-5dabc0d1-cjzsv\" (UID: \"9caa8f05-7487-4eec-87fb-cdb3b85d5f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"
Nov 02 12:18:19.836569 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.836549    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dabc0d1-cjzsv\" (UID: \"9caa8f05-7487-4eec-87fb-cdb3b85d5f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"
Nov 02 12:18:19.836632 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.836616    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dabc0d1-cjzsv\" (UID: \"9caa8f05-7487-4eec-87fb-cdb3b85d5f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"
Nov 02 12:18:19.937525 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.937492    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dabc0d1-cjzsv\" (UID: \"9caa8f05-7487-4eec-87fb-cdb3b85d5f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"
Nov 02 12:18:19.937679 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.937558    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dabc0d1-cjzsv\" (UID: \"9caa8f05-7487-4eec-87fb-cdb3b85d5f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"
Nov 02 12:18:19.937679 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.937616    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm78x\" (UniqueName: \"kubernetes.io/projected/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-kube-api-access-fm78x\") pod \"maas-api-revoke-sub-5dabc0d1-cjzsv\" (UID: \"9caa8f05-7487-4eec-87fb-cdb3b85d5f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"
Nov 02 12:18:19.938117 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.938097    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dabc0d1-cjzsv\" (UID: \"9caa8f05-7487-4eec-87fb-cdb3b85d5f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"
Nov 02 12:18:19.939830 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.939805    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dabc0d1-cjzsv\" (UID: \"9caa8f05-7487-4eec-87fb-cdb3b85d5f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"
Nov 02 12:18:20.000068 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:19.999997    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm78x\" (UniqueName: \"kubernetes.io/projected/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-kube-api-access-fm78x\") pod \"maas-api-revoke-sub-5dabc0d1-cjzsv\" (UID: \"9caa8f05-7487-4eec-87fb-cdb3b85d5f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"
Nov 02 12:18:20.026868 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:20.026841    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"
Nov 02 12:18:20.406729 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:18:20.406692    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9caa8f05_7487_4eec_87fb_cdb3b85d5f96.slice/crio-f5765ffa1c058b5648a26520ec73c4651527bd0f716180b2efdbe6cc476e55d8 WatchSource:0}: Error finding container f5765ffa1c058b5648a26520ec73c4651527bd0f716180b2efdbe6cc476e55d8: Status 404 returned error can't find the container with id f5765ffa1c058b5648a26520ec73c4651527bd0f716180b2efdbe6cc476e55d8
Nov 02 12:18:20.407757 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:20.407732    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"]
Nov 02 12:18:21.007564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:21.007531    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv" event={"ID":"9caa8f05-7487-4eec-87fb-cdb3b85d5f96","Type":"ContainerStarted","Data":"4eba4344af5b9231041c889af30ca39d23d2255a525f3b66cefb69ca4a3754d2"}
Nov 02 12:18:21.007918 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:21.007585    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv" event={"ID":"9caa8f05-7487-4eec-87fb-cdb3b85d5f96","Type":"ContainerStarted","Data":"f5765ffa1c058b5648a26520ec73c4651527bd0f716180b2efdbe6cc476e55d8"}
Nov 02 12:18:21.131421 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:21.131371    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv" podStartSLOduration=2.131358304 podStartE2EDuration="2.131358304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:18:21.128104306 +0000 UTC m=+1329.691812717" watchObservedRunningTime="2026-11-02 12:18:21.131358304 +0000 UTC m=+1329.695066714"
Nov 02 12:18:22.015628 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:22.015591    4464 generic.go:356] "Generic (PLEG): container finished" podID="9caa8f05-7487-4eec-87fb-cdb3b85d5f96" containerID="4eba4344af5b9231041c889af30ca39d23d2255a525f3b66cefb69ca4a3754d2" exitCode=0
Nov 02 12:18:22.016030 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:22.015674    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv" event={"ID":"9caa8f05-7487-4eec-87fb-cdb3b85d5f96","Type":"ContainerDied","Data":"4eba4344af5b9231041c889af30ca39d23d2255a525f3b66cefb69ca4a3754d2"}
Nov 02 12:18:23.176623 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:23.176597    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"
Nov 02 12:18:23.288950 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:23.288872    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-maas-api-xks-ca\") pod \"9caa8f05-7487-4eec-87fb-cdb3b85d5f96\" (UID: \"9caa8f05-7487-4eec-87fb-cdb3b85d5f96\") "
Nov 02 12:18:23.289119 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:23.289055    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-kube-api-access-fm78x\" (UniqueName: \"kubernetes.io/projected/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-kube-api-access-fm78x\") pod \"9caa8f05-7487-4eec-87fb-cdb3b85d5f96\" (UID: \"9caa8f05-7487-4eec-87fb-cdb3b85d5f96\") "
Nov 02 12:18:23.289195 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:23.289136    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-maas-api-openshift-ca\") pod \"9caa8f05-7487-4eec-87fb-cdb3b85d5f96\" (UID: \"9caa8f05-7487-4eec-87fb-cdb3b85d5f96\") "
Nov 02 12:18:23.289485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:23.289463    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-maas-api-openshift-ca" pod "9caa8f05-7487-4eec-87fb-cdb3b85d5f96" (UID: "9caa8f05-7487-4eec-87fb-cdb3b85d5f96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:18:23.290943 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:23.290912    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-maas-api-xks-ca" pod "9caa8f05-7487-4eec-87fb-cdb3b85d5f96" (UID: "9caa8f05-7487-4eec-87fb-cdb3b85d5f96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:18:23.291066 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:23.291046    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-kube-api-access-fm78x" pod "9caa8f05-7487-4eec-87fb-cdb3b85d5f96" (UID: "9caa8f05-7487-4eec-87fb-cdb3b85d5f96"). InnerVolumeSpecName "kube-api-access-fm78x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:18:23.390606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:23.390569    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm78x\" (UniqueName: \"kubernetes.io/projected/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-kube-api-access-fm78x\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:23.390606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:23.390601    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:23.390823 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:23.390614    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9caa8f05-7487-4eec-87fb-cdb3b85d5f96-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:24.033788 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:24.033760    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"
Nov 02 12:18:24.033954 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:24.033796    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv" event={"ID":"9caa8f05-7487-4eec-87fb-cdb3b85d5f96","Type":"ContainerDied","Data":"f5765ffa1c058b5648a26520ec73c4651527bd0f716180b2efdbe6cc476e55d8"}
Nov 02 12:18:24.033954 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:24.033845    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5765ffa1c058b5648a26520ec73c4651527bd0f716180b2efdbe6cc476e55d8"
Nov 02 12:18:28.349429 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:18:28.349386    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:18:30.452370 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.447492    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"]
Nov 02 12:18:30.452370 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.448902    4464 cpu_manager.go:414] "containerMap: removing container" podUID="9caa8f05-7487-4eec-87fb-cdb3b85d5f96" containerName="revoke-keys"
Nov 02 12:18:30.452370 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.448926    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9caa8f05-7487-4eec-87fb-cdb3b85d5f96" containerName="revoke-keys"
Nov 02 12:18:30.452370 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.449322    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="9caa8f05-7487-4eec-87fb-cdb3b85d5f96" containerName="revoke-keys"
Nov 02 12:18:30.455106 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.455075    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"
Nov 02 12:18:30.508557 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.508528    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"]
Nov 02 12:18:30.546069 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.546042    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fd29b28-mt8w7\" (UID: \"16f72bca-a91b-41d8-b3e7-764c9e53ad5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"
Nov 02 12:18:30.546214 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.546177    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fd29b28-mt8w7\" (UID: \"16f72bca-a91b-41d8-b3e7-764c9e53ad5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"
Nov 02 12:18:30.546351 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.546337    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgpps\" (UniqueName: \"kubernetes.io/projected/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-kube-api-access-wgpps\") pod \"maas-api-revoke-sub-3fd29b28-mt8w7\" (UID: \"16f72bca-a91b-41d8-b3e7-764c9e53ad5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"
Nov 02 12:18:30.552507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.552484    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:18:30.552507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.552502    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:18:30.552662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.552514    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:18:30.558959 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.558943    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:18:30.647186 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.647146    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fd29b28-mt8w7\" (UID: \"16f72bca-a91b-41d8-b3e7-764c9e53ad5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"
Nov 02 12:18:30.647352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.647248    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fd29b28-mt8w7\" (UID: \"16f72bca-a91b-41d8-b3e7-764c9e53ad5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"
Nov 02 12:18:30.647433 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.647414    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgpps\" (UniqueName: \"kubernetes.io/projected/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-kube-api-access-wgpps\") pod \"maas-api-revoke-sub-3fd29b28-mt8w7\" (UID: \"16f72bca-a91b-41d8-b3e7-764c9e53ad5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"
Nov 02 12:18:30.647887 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.647867    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fd29b28-mt8w7\" (UID: \"16f72bca-a91b-41d8-b3e7-764c9e53ad5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"
Nov 02 12:18:30.649678 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.649655    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fd29b28-mt8w7\" (UID: \"16f72bca-a91b-41d8-b3e7-764c9e53ad5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"
Nov 02 12:18:30.709652 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.709590    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgpps\" (UniqueName: \"kubernetes.io/projected/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-kube-api-access-wgpps\") pod \"maas-api-revoke-sub-3fd29b28-mt8w7\" (UID: \"16f72bca-a91b-41d8-b3e7-764c9e53ad5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"
Nov 02 12:18:30.767953 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:30.767923    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"
Nov 02 12:18:31.144721 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:18:31.144692    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f72bca_a91b_41d8_b3e7_764c9e53ad5f.slice/crio-2cd7722666f3d4b9fc04f7352078e4102a846501a185f28d9770e270c5f0d3e0 WatchSource:0}: Error finding container 2cd7722666f3d4b9fc04f7352078e4102a846501a185f28d9770e270c5f0d3e0: Status 404 returned error can't find the container with id 2cd7722666f3d4b9fc04f7352078e4102a846501a185f28d9770e270c5f0d3e0
Nov 02 12:18:31.146284 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:31.146258    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"]
Nov 02 12:18:32.134661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:32.134614    4464 generic.go:356] "Generic (PLEG): container finished" podID="16f72bca-a91b-41d8-b3e7-764c9e53ad5f" containerID="3212566d8920e7ad968e770cdbcb4da810c3c778991b81fdc9c4249f5c57b171" exitCode=0
Nov 02 12:18:32.134661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:32.134653    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7" event={"ID":"16f72bca-a91b-41d8-b3e7-764c9e53ad5f","Type":"ContainerDied","Data":"3212566d8920e7ad968e770cdbcb4da810c3c778991b81fdc9c4249f5c57b171"}
Nov 02 12:18:32.135170 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:32.134714    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7" event={"ID":"16f72bca-a91b-41d8-b3e7-764c9e53ad5f","Type":"ContainerStarted","Data":"2cd7722666f3d4b9fc04f7352078e4102a846501a185f28d9770e270c5f0d3e0"}
Nov 02 12:18:33.132531 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:33.132494    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"]
Nov 02 12:18:33.384787 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:33.384732    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"
Nov 02 12:18:33.491675 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:33.491645    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-maas-api-xks-ca\") pod \"16f72bca-a91b-41d8-b3e7-764c9e53ad5f\" (UID: \"16f72bca-a91b-41d8-b3e7-764c9e53ad5f\") "
Nov 02 12:18:33.491825 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:33.491697    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-kube-api-access-wgpps\" (UniqueName: \"kubernetes.io/projected/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-kube-api-access-wgpps\") pod \"16f72bca-a91b-41d8-b3e7-764c9e53ad5f\" (UID: \"16f72bca-a91b-41d8-b3e7-764c9e53ad5f\") "
Nov 02 12:18:33.491825 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:33.491759    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-maas-api-openshift-ca\") pod \"16f72bca-a91b-41d8-b3e7-764c9e53ad5f\" (UID: \"16f72bca-a91b-41d8-b3e7-764c9e53ad5f\") "
Nov 02 12:18:33.492175 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:33.492150    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-maas-api-openshift-ca" pod "16f72bca-a91b-41d8-b3e7-764c9e53ad5f" (UID: "16f72bca-a91b-41d8-b3e7-764c9e53ad5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:18:33.493757 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:33.493736    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-kube-api-access-wgpps" pod "16f72bca-a91b-41d8-b3e7-764c9e53ad5f" (UID: "16f72bca-a91b-41d8-b3e7-764c9e53ad5f"). InnerVolumeSpecName "kube-api-access-wgpps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:18:33.493757 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:33.493741    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-maas-api-xks-ca" pod "16f72bca-a91b-41d8-b3e7-764c9e53ad5f" (UID: "16f72bca-a91b-41d8-b3e7-764c9e53ad5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:18:33.593427 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:33.593399    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:33.593427 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:33.593426    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgpps\" (UniqueName: \"kubernetes.io/projected/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-kube-api-access-wgpps\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:33.593581 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:33.593442    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f72bca-a91b-41d8-b3e7-764c9e53ad5f-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:34.151156 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:34.151123    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"
Nov 02 12:18:34.151156 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:34.151137    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7" event={"ID":"16f72bca-a91b-41d8-b3e7-764c9e53ad5f","Type":"ContainerDied","Data":"2cd7722666f3d4b9fc04f7352078e4102a846501a185f28d9770e270c5f0d3e0"}
Nov 02 12:18:34.151424 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:34.151208    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cd7722666f3d4b9fc04f7352078e4102a846501a185f28d9770e270c5f0d3e0"
Nov 02 12:18:35.049895 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:35.049834    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"]
Nov 02 12:18:35.102109 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:35.102081    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95a16573-mvj9z"]
Nov 02 12:18:35.583246 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:35.583214    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b49c47f0-83dc-4fff-b9ed-01136280bbfe" path="/var/lib/kubelet/pods/b49c47f0-83dc-4fff-b9ed-01136280bbfe/volumes"
Nov 02 12:18:39.846943 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:39.846909    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:18:46.049866 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:46.049830    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"]
Nov 02 12:18:46.100264 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:46.100232    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047c00af-z7nq8"]
Nov 02 12:18:47.583033 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:47.582971    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a738826-ed52-4e69-8623-04409ac092df" path="/var/lib/kubelet/pods/4a738826-ed52-4e69-8623-04409ac092df/volumes"
Nov 02 12:18:48.153679 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.153635    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72" podUID="df859e10-37e4-4d2e-a1d5-931adacbaaff" containerName="main" containerID="cri-o://c56c6bd30b55f719bab89f1de8076bdec8dcfe67936519d1d3abc20fc95825fd" gracePeriod=45
Nov 02 12:18:48.358288 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.358253    4464 generic.go:356] "Generic (PLEG): container finished" podID="df859e10-37e4-4d2e-a1d5-931adacbaaff" containerID="c56c6bd30b55f719bab89f1de8076bdec8dcfe67936519d1d3abc20fc95825fd" exitCode=0
Nov 02 12:18:48.358463 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.358422    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72" event={"ID":"df859e10-37e4-4d2e-a1d5-931adacbaaff","Type":"ContainerDied","Data":"c56c6bd30b55f719bab89f1de8076bdec8dcfe67936519d1d3abc20fc95825fd"}
Nov 02 12:18:48.429199 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.429170    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:18:48.507332 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.507302    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-tmp-dir\") pod \"df859e10-37e4-4d2e-a1d5-931adacbaaff\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") "
Nov 02 12:18:48.507515 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.507377    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df859e10-37e4-4d2e-a1d5-931adacbaaff-tls-certs\" (UniqueName: \"kubernetes.io/secret/df859e10-37e4-4d2e-a1d5-931adacbaaff-tls-certs\") pod \"df859e10-37e4-4d2e-a1d5-931adacbaaff\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") "
Nov 02 12:18:48.507515 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.507417    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-dshm\") pod \"df859e10-37e4-4d2e-a1d5-931adacbaaff\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") "
Nov 02 12:18:48.507515 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.507440    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-home\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-home\") pod \"df859e10-37e4-4d2e-a1d5-931adacbaaff\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") "
Nov 02 12:18:48.507677 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.507532    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-model-cache\") pod \"df859e10-37e4-4d2e-a1d5-931adacbaaff\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") "
Nov 02 12:18:48.507677 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.507545    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-tmp-dir" pod "df859e10-37e4-4d2e-a1d5-931adacbaaff" (UID: "df859e10-37e4-4d2e-a1d5-931adacbaaff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:18:48.507677 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.507667    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df859e10-37e4-4d2e-a1d5-931adacbaaff-kube-api-access-g2njj\" (UniqueName: \"kubernetes.io/projected/df859e10-37e4-4d2e-a1d5-931adacbaaff-kube-api-access-g2njj\") pod \"df859e10-37e4-4d2e-a1d5-931adacbaaff\" (UID: \"df859e10-37e4-4d2e-a1d5-931adacbaaff\") "
Nov 02 12:18:48.507823 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.507737    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-home" pod "df859e10-37e4-4d2e-a1d5-931adacbaaff" (UID: "df859e10-37e4-4d2e-a1d5-931adacbaaff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:18:48.507823 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.507774    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-model-cache" pod "df859e10-37e4-4d2e-a1d5-931adacbaaff" (UID: "df859e10-37e4-4d2e-a1d5-931adacbaaff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:18:48.508171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.508153    4464 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-model-cache\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:48.508259 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.508173    4464 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-tmp-dir\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:48.508259 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.508185    4464 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-home\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:48.509634 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.509609    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df859e10-37e4-4d2e-a1d5-931adacbaaff-tls-certs" pod "df859e10-37e4-4d2e-a1d5-931adacbaaff" (UID: "df859e10-37e4-4d2e-a1d5-931adacbaaff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:18:48.509934 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.509907    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-dshm" pod "df859e10-37e4-4d2e-a1d5-931adacbaaff" (UID: "df859e10-37e4-4d2e-a1d5-931adacbaaff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:18:48.510059 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.509946    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df859e10-37e4-4d2e-a1d5-931adacbaaff-kube-api-access-g2njj" pod "df859e10-37e4-4d2e-a1d5-931adacbaaff" (UID: "df859e10-37e4-4d2e-a1d5-931adacbaaff"). InnerVolumeSpecName "kube-api-access-g2njj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:18:48.609847 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.609819    4464 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df859e10-37e4-4d2e-a1d5-931adacbaaff-tls-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:48.609847 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.609842    4464 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df859e10-37e4-4d2e-a1d5-931adacbaaff-dshm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:48.609847 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.609855    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2njj\" (UniqueName: \"kubernetes.io/projected/df859e10-37e4-4d2e-a1d5-931adacbaaff-kube-api-access-g2njj\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:48.831006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.827686    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"]
Nov 02 12:18:48.831006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.829148    4464 cpu_manager.go:414] "containerMap: removing container" podUID="16f72bca-a91b-41d8-b3e7-764c9e53ad5f" containerName="revoke-keys"
Nov 02 12:18:48.831006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.829169    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f72bca-a91b-41d8-b3e7-764c9e53ad5f" containerName="revoke-keys"
Nov 02 12:18:48.831006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.829192    4464 cpu_manager.go:414] "containerMap: removing container" podUID="df859e10-37e4-4d2e-a1d5-931adacbaaff" containerName="main"
Nov 02 12:18:48.831006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.829199    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df859e10-37e4-4d2e-a1d5-931adacbaaff" containerName="main"
Nov 02 12:18:48.831006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.829482    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="16f72bca-a91b-41d8-b3e7-764c9e53ad5f" containerName="revoke-keys"
Nov 02 12:18:48.831006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.829504    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="df859e10-37e4-4d2e-a1d5-931adacbaaff" containerName="main"
Nov 02 12:18:48.836096 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.836049    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"
Nov 02 12:18:48.888419 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.888384    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"]
Nov 02 12:18:48.896245 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.896218    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:18:48.913174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.913145    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07a2c897-xrml8\" (UID: \"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"
Nov 02 12:18:48.913314 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.913274    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ds66\" (UniqueName: \"kubernetes.io/projected/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-kube-api-access-4ds66\") pod \"maas-api-revoke-sub-07a2c897-xrml8\" (UID: \"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"
Nov 02 12:18:48.913408 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.913392    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07a2c897-xrml8\" (UID: \"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"
Nov 02 12:18:48.931922 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.931875    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:18:48.931922 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.931908    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:18:48.932115 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:48.932051    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:18:49.014778 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:49.014736    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ds66\" (UniqueName: \"kubernetes.io/projected/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-kube-api-access-4ds66\") pod \"maas-api-revoke-sub-07a2c897-xrml8\" (UID: \"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"
Nov 02 12:18:49.014971 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:49.014832    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07a2c897-xrml8\" (UID: \"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"
Nov 02 12:18:49.014971 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:49.014968    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07a2c897-xrml8\" (UID: \"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"
Nov 02 12:18:49.015459 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:49.015437    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07a2c897-xrml8\" (UID: \"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"
Nov 02 12:18:49.017319 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:49.017296    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07a2c897-xrml8\" (UID: \"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"
Nov 02 12:18:49.076522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:49.076489    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ds66\" (UniqueName: \"kubernetes.io/projected/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-kube-api-access-4ds66\") pod \"maas-api-revoke-sub-07a2c897-xrml8\" (UID: \"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"
Nov 02 12:18:49.156318 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:49.156240    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"
Nov 02 12:18:49.367918 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:49.367888    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"
Nov 02 12:18:49.368106 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:49.367909    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72" event={"ID":"df859e10-37e4-4d2e-a1d5-931adacbaaff","Type":"ContainerDied","Data":"0c129eafe51b6e5793e3ee5d0a6b92ef3fafe65cebcb97f6b870563fa8d70dad"}
Nov 02 12:18:49.368106 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:49.367962    4464 scope.go:122] "RemoveContainer" containerID="c56c6bd30b55f719bab89f1de8076bdec8dcfe67936519d1d3abc20fc95825fd"
Nov 02 12:18:49.537629 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:49.537598    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"]
Nov 02 12:18:49.541693 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:49.541669    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"]
Nov 02 12:18:49.579094 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:18:49.579065    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2d46c4_eb23_4706_93d6_e9f3f4facd2a.slice/crio-5a33dbbd5c960a36270a1611c2de1d90f74da3febc9404307e629bc2f110fe00 WatchSource:0}: Error finding container 5a33dbbd5c960a36270a1611c2de1d90f74da3febc9404307e629bc2f110fe00: Status 404 returned error can't find the container with id 5a33dbbd5c960a36270a1611c2de1d90f74da3febc9404307e629bc2f110fe00
Nov 02 12:18:49.587546 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:49.587524    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-036f88/test-model-036f88-kserve-766b784d9c-fcw72"]
Nov 02 12:18:50.377053 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:50.377010    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8" event={"ID":"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a","Type":"ContainerStarted","Data":"128df30c7bfd99963e7f729fc79274fd6393101d08e8d7a9dbe61da9f8b77226"}
Nov 02 12:18:50.377516 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:50.377077    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8" event={"ID":"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a","Type":"ContainerStarted","Data":"5a33dbbd5c960a36270a1611c2de1d90f74da3febc9404307e629bc2f110fe00"}
Nov 02 12:18:50.495646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:50.495596    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8" podStartSLOduration=2.4955824939999998 podStartE2EDuration="2.495582494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:18:50.494652189 +0000 UTC m=+1359.058360598" watchObservedRunningTime="2026-11-02 12:18:50.495582494 +0000 UTC m=+1359.059290902"
Nov 02 12:18:51.386468 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:51.386435    4464 generic.go:356] "Generic (PLEG): container finished" podID="4a2d46c4-eb23-4706-93d6-e9f3f4facd2a" containerID="128df30c7bfd99963e7f729fc79274fd6393101d08e8d7a9dbe61da9f8b77226" exitCode=0
Nov 02 12:18:51.386893 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:51.386492    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8" event={"ID":"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a","Type":"ContainerDied","Data":"128df30c7bfd99963e7f729fc79274fd6393101d08e8d7a9dbe61da9f8b77226"}
Nov 02 12:18:51.582712 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:51.582680    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df859e10-37e4-4d2e-a1d5-931adacbaaff" path="/var/lib/kubelet/pods/df859e10-37e4-4d2e-a1d5-931adacbaaff/volumes"
Nov 02 12:18:52.551987 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:52.551948    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"
Nov 02 12:18:52.666606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:52.666525    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-kube-api-access-4ds66\" (UniqueName: \"kubernetes.io/projected/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-kube-api-access-4ds66\") pod \"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a\" (UID: \"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a\") "
Nov 02 12:18:52.666606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:52.666594    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-maas-api-xks-ca\") pod \"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a\" (UID: \"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a\") "
Nov 02 12:18:52.666833 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:52.666654    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-maas-api-openshift-ca\") pod \"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a\" (UID: \"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a\") "
Nov 02 12:18:52.667098 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:52.667063    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-maas-api-openshift-ca" pod "4a2d46c4-eb23-4706-93d6-e9f3f4facd2a" (UID: "4a2d46c4-eb23-4706-93d6-e9f3f4facd2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:18:52.668579 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:52.668556    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-maas-api-xks-ca" pod "4a2d46c4-eb23-4706-93d6-e9f3f4facd2a" (UID: "4a2d46c4-eb23-4706-93d6-e9f3f4facd2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:18:52.668669 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:52.668633    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-kube-api-access-4ds66" pod "4a2d46c4-eb23-4706-93d6-e9f3f4facd2a" (UID: "4a2d46c4-eb23-4706-93d6-e9f3f4facd2a"). InnerVolumeSpecName "kube-api-access-4ds66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:18:52.768320 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:52.768275    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:52.768320 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:52.768315    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ds66\" (UniqueName: \"kubernetes.io/projected/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-kube-api-access-4ds66\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:52.768320 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:52.768329    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:18:53.403061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:53.403037    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"
Nov 02 12:18:53.403222 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:53.403036    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8" event={"ID":"4a2d46c4-eb23-4706-93d6-e9f3f4facd2a","Type":"ContainerDied","Data":"5a33dbbd5c960a36270a1611c2de1d90f74da3febc9404307e629bc2f110fe00"}
Nov 02 12:18:53.403222 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:53.403178    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a33dbbd5c960a36270a1611c2de1d90f74da3febc9404307e629bc2f110fe00"
Nov 02 12:18:58.291878 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:18:58.291844    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:18:59.632998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.628705    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"]
Nov 02 12:18:59.632998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.630409    4464 cpu_manager.go:414] "containerMap: removing container" podUID="4a2d46c4-eb23-4706-93d6-e9f3f4facd2a" containerName="revoke-keys"
Nov 02 12:18:59.632998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.630430    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a2d46c4-eb23-4706-93d6-e9f3f4facd2a" containerName="revoke-keys"
Nov 02 12:18:59.632998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.630576    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a2d46c4-eb23-4706-93d6-e9f3f4facd2a" containerName="revoke-keys"
Nov 02 12:18:59.635058 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.635033    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"
Nov 02 12:18:59.644543 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.644505    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"]
Nov 02 12:18:59.742458 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.742411    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:18:59.742645 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.742501    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:18:59.742645 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.742546    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:18:59.742645 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.742634    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:18:59.752163 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.752126    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acf9653-c380-411c-a0a3-18f49e9f2c9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adbfad1a-msskc\" (UID: \"4acf9653-c380-411c-a0a3-18f49e9f2c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"
Nov 02 12:18:59.752334 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.752308    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acf9653-c380-411c-a0a3-18f49e9f2c9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adbfad1a-msskc\" (UID: \"4acf9653-c380-411c-a0a3-18f49e9f2c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"
Nov 02 12:18:59.752506 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.752486    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b8m6\" (UniqueName: \"kubernetes.io/projected/4acf9653-c380-411c-a0a3-18f49e9f2c9e-kube-api-access-8b8m6\") pod \"maas-api-revoke-sub-adbfad1a-msskc\" (UID: \"4acf9653-c380-411c-a0a3-18f49e9f2c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"
Nov 02 12:18:59.853415 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.853382    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acf9653-c380-411c-a0a3-18f49e9f2c9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adbfad1a-msskc\" (UID: \"4acf9653-c380-411c-a0a3-18f49e9f2c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"
Nov 02 12:18:59.853592 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.853533    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b8m6\" (UniqueName: \"kubernetes.io/projected/4acf9653-c380-411c-a0a3-18f49e9f2c9e-kube-api-access-8b8m6\") pod \"maas-api-revoke-sub-adbfad1a-msskc\" (UID: \"4acf9653-c380-411c-a0a3-18f49e9f2c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"
Nov 02 12:18:59.853660 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.853594    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acf9653-c380-411c-a0a3-18f49e9f2c9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adbfad1a-msskc\" (UID: \"4acf9653-c380-411c-a0a3-18f49e9f2c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"
Nov 02 12:18:59.854271 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.854242    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acf9653-c380-411c-a0a3-18f49e9f2c9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adbfad1a-msskc\" (UID: \"4acf9653-c380-411c-a0a3-18f49e9f2c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"
Nov 02 12:18:59.856081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.856052    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acf9653-c380-411c-a0a3-18f49e9f2c9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adbfad1a-msskc\" (UID: \"4acf9653-c380-411c-a0a3-18f49e9f2c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"
Nov 02 12:18:59.925508 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:18:59.925385    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b8m6\" (UniqueName: \"kubernetes.io/projected/4acf9653-c380-411c-a0a3-18f49e9f2c9e-kube-api-access-8b8m6\") pod \"maas-api-revoke-sub-adbfad1a-msskc\" (UID: \"4acf9653-c380-411c-a0a3-18f49e9f2c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"
Nov 02 12:19:00.005845 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:00.005523    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"
Nov 02 12:19:00.632002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:00.631891    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"]
Nov 02 12:19:00.832991 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:00.832930    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"]
Nov 02 12:19:00.875604 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:00.874622    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"
Nov 02 12:19:01.098000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:01.036474    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"]
Nov 02 12:19:01.417001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:01.416744    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn9fr\" (UniqueName: \"kubernetes.io/projected/feb5d296-405a-4255-812b-8b6f1005b0c3-kube-api-access-jn9fr\") pod \"maas-api-revoke-sub-054acce8-lwsn8\" (UID: \"feb5d296-405a-4255-812b-8b6f1005b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"
Nov 02 12:19:01.417001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:01.416832    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb5d296-405a-4255-812b-8b6f1005b0c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-054acce8-lwsn8\" (UID: \"feb5d296-405a-4255-812b-8b6f1005b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"
Nov 02 12:19:01.417001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:01.416928    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb5d296-405a-4255-812b-8b6f1005b0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-054acce8-lwsn8\" (UID: \"feb5d296-405a-4255-812b-8b6f1005b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"
Nov 02 12:19:01.530360 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:01.522819    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb5d296-405a-4255-812b-8b6f1005b0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-054acce8-lwsn8\" (UID: \"feb5d296-405a-4255-812b-8b6f1005b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"
Nov 02 12:19:01.530360 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:01.523062    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn9fr\" (UniqueName: \"kubernetes.io/projected/feb5d296-405a-4255-812b-8b6f1005b0c3-kube-api-access-jn9fr\") pod \"maas-api-revoke-sub-054acce8-lwsn8\" (UID: \"feb5d296-405a-4255-812b-8b6f1005b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"
Nov 02 12:19:01.530360 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:01.523107    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb5d296-405a-4255-812b-8b6f1005b0c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-054acce8-lwsn8\" (UID: \"feb5d296-405a-4255-812b-8b6f1005b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"
Nov 02 12:19:01.530360 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:01.525815    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb5d296-405a-4255-812b-8b6f1005b0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-054acce8-lwsn8\" (UID: \"feb5d296-405a-4255-812b-8b6f1005b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"
Nov 02 12:19:01.559129 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:01.559095    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb5d296-405a-4255-812b-8b6f1005b0c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-054acce8-lwsn8\" (UID: \"feb5d296-405a-4255-812b-8b6f1005b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"
Nov 02 12:19:01.644999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:01.638151    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn9fr\" (UniqueName: \"kubernetes.io/projected/feb5d296-405a-4255-812b-8b6f1005b0c3-kube-api-access-jn9fr\") pod \"maas-api-revoke-sub-054acce8-lwsn8\" (UID: \"feb5d296-405a-4255-812b-8b6f1005b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"
Nov 02 12:19:01.742001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:01.728844    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc" event={"ID":"4acf9653-c380-411c-a0a3-18f49e9f2c9e","Type":"ContainerStarted","Data":"02bc30bb9426ada9eaa930939ec2dc8ae5a2ccab9aa49ad277e7bf058bba6ebd"}
Nov 02 12:19:01.822222 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:01.822180    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"
Nov 02 12:19:02.840004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:02.839020    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc" event={"ID":"4acf9653-c380-411c-a0a3-18f49e9f2c9e","Type":"ContainerStarted","Data":"2887d628f26e26e236fd7eb89f2f199be7917f0fba3a3f4ab1a076cb376d10ff"}
Nov 02 12:19:03.069534 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:03.069501    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"]
Nov 02 12:19:03.849057 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:03.849021    4464 generic.go:356] "Generic (PLEG): container finished" podID="4acf9653-c380-411c-a0a3-18f49e9f2c9e" containerID="2887d628f26e26e236fd7eb89f2f199be7917f0fba3a3f4ab1a076cb376d10ff" exitCode=0
Nov 02 12:19:03.849445 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:03.849105    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc" event={"ID":"4acf9653-c380-411c-a0a3-18f49e9f2c9e","Type":"ContainerDied","Data":"2887d628f26e26e236fd7eb89f2f199be7917f0fba3a3f4ab1a076cb376d10ff"}
Nov 02 12:19:03.850902 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:03.850873    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8" event={"ID":"feb5d296-405a-4255-812b-8b6f1005b0c3","Type":"ContainerStarted","Data":"33537237feeb0a10977bd5f2283e05516d59ae62694dd39211d121d37c841b14"}
Nov 02 12:19:03.851060 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:03.850920    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8" event={"ID":"feb5d296-405a-4255-812b-8b6f1005b0c3","Type":"ContainerStarted","Data":"647175c3f9ec84e5bb8b7a2a0f24cf9d94c51c8e7ba0b9a90af0fde0df04ff37"}
Nov 02 12:19:03.970658 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:03.970607    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8" podStartSLOduration=3.9705890090000002 podStartE2EDuration="3.970589009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:19:03.968730839 +0000 UTC m=+1372.532439247" watchObservedRunningTime="2026-11-02 12:19:03.970589009 +0000 UTC m=+1372.534297395"
Nov 02 12:19:04.049946 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:04.049923    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"
Nov 02 12:19:04.070833 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:04.070809    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4acf9653-c380-411c-a0a3-18f49e9f2c9e-kube-api-access-8b8m6\" (UniqueName: \"kubernetes.io/projected/4acf9653-c380-411c-a0a3-18f49e9f2c9e-kube-api-access-8b8m6\") pod \"4acf9653-c380-411c-a0a3-18f49e9f2c9e\" (UID: \"4acf9653-c380-411c-a0a3-18f49e9f2c9e\") "
Nov 02 12:19:04.070950 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:04.070878    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4acf9653-c380-411c-a0a3-18f49e9f2c9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acf9653-c380-411c-a0a3-18f49e9f2c9e-maas-api-openshift-ca\") pod \"4acf9653-c380-411c-a0a3-18f49e9f2c9e\" (UID: \"4acf9653-c380-411c-a0a3-18f49e9f2c9e\") "
Nov 02 12:19:04.071057 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:04.071040    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4acf9653-c380-411c-a0a3-18f49e9f2c9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acf9653-c380-411c-a0a3-18f49e9f2c9e-maas-api-xks-ca\") pod \"4acf9653-c380-411c-a0a3-18f49e9f2c9e\" (UID: \"4acf9653-c380-411c-a0a3-18f49e9f2c9e\") "
Nov 02 12:19:04.071171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:04.071153    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4acf9653-c380-411c-a0a3-18f49e9f2c9e-maas-api-openshift-ca" pod "4acf9653-c380-411c-a0a3-18f49e9f2c9e" (UID: "4acf9653-c380-411c-a0a3-18f49e9f2c9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:19:04.071552 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:04.071529    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4acf9653-c380-411c-a0a3-18f49e9f2c9e-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:04.072941 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:04.072917    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4acf9653-c380-411c-a0a3-18f49e9f2c9e-maas-api-xks-ca" pod "4acf9653-c380-411c-a0a3-18f49e9f2c9e" (UID: "4acf9653-c380-411c-a0a3-18f49e9f2c9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:19:04.073054 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:04.072955    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4acf9653-c380-411c-a0a3-18f49e9f2c9e-kube-api-access-8b8m6" pod "4acf9653-c380-411c-a0a3-18f49e9f2c9e" (UID: "4acf9653-c380-411c-a0a3-18f49e9f2c9e"). InnerVolumeSpecName "kube-api-access-8b8m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:19:04.172722 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:04.172652    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4acf9653-c380-411c-a0a3-18f49e9f2c9e-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:04.172722 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:04.172684    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b8m6\" (UniqueName: \"kubernetes.io/projected/4acf9653-c380-411c-a0a3-18f49e9f2c9e-kube-api-access-8b8m6\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:04.859143 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:04.859115    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"
Nov 02 12:19:04.859583 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:04.859156    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc" event={"ID":"4acf9653-c380-411c-a0a3-18f49e9f2c9e","Type":"ContainerDied","Data":"02bc30bb9426ada9eaa930939ec2dc8ae5a2ccab9aa49ad277e7bf058bba6ebd"}
Nov 02 12:19:04.859583 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:04.859202    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02bc30bb9426ada9eaa930939ec2dc8ae5a2ccab9aa49ad277e7bf058bba6ebd"
Nov 02 12:19:04.860938 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:04.860919    4464 generic.go:356] "Generic (PLEG): container finished" podID="feb5d296-405a-4255-812b-8b6f1005b0c3" containerID="33537237feeb0a10977bd5f2283e05516d59ae62694dd39211d121d37c841b14" exitCode=0
Nov 02 12:19:04.861050 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:04.861001    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8" event={"ID":"feb5d296-405a-4255-812b-8b6f1005b0c3","Type":"ContainerDied","Data":"33537237feeb0a10977bd5f2283e05516d59ae62694dd39211d121d37c841b14"}
Nov 02 12:19:06.022065 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:06.022036    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"
Nov 02 12:19:06.094019 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:06.093957    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feb5d296-405a-4255-812b-8b6f1005b0c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb5d296-405a-4255-812b-8b6f1005b0c3-maas-api-openshift-ca\") pod \"feb5d296-405a-4255-812b-8b6f1005b0c3\" (UID: \"feb5d296-405a-4255-812b-8b6f1005b0c3\") "
Nov 02 12:19:06.094140 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:06.094067    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb5d296-405a-4255-812b-8b6f1005b0c3-kube-api-access-jn9fr\" (UniqueName: \"kubernetes.io/projected/feb5d296-405a-4255-812b-8b6f1005b0c3-kube-api-access-jn9fr\") pod \"feb5d296-405a-4255-812b-8b6f1005b0c3\" (UID: \"feb5d296-405a-4255-812b-8b6f1005b0c3\") "
Nov 02 12:19:06.094140 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:06.094111    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feb5d296-405a-4255-812b-8b6f1005b0c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb5d296-405a-4255-812b-8b6f1005b0c3-maas-api-xks-ca\") pod \"feb5d296-405a-4255-812b-8b6f1005b0c3\" (UID: \"feb5d296-405a-4255-812b-8b6f1005b0c3\") "
Nov 02 12:19:06.094323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:06.094302    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb5d296-405a-4255-812b-8b6f1005b0c3-maas-api-openshift-ca" pod "feb5d296-405a-4255-812b-8b6f1005b0c3" (UID: "feb5d296-405a-4255-812b-8b6f1005b0c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:19:06.094618 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:06.094594    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feb5d296-405a-4255-812b-8b6f1005b0c3-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:06.096555 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:06.096527    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feb5d296-405a-4255-812b-8b6f1005b0c3-maas-api-xks-ca" pod "feb5d296-405a-4255-812b-8b6f1005b0c3" (UID: "feb5d296-405a-4255-812b-8b6f1005b0c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:19:06.096766 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:06.096744    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb5d296-405a-4255-812b-8b6f1005b0c3-kube-api-access-jn9fr" pod "feb5d296-405a-4255-812b-8b6f1005b0c3" (UID: "feb5d296-405a-4255-812b-8b6f1005b0c3"). InnerVolumeSpecName "kube-api-access-jn9fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:19:06.195995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:06.195911    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn9fr\" (UniqueName: \"kubernetes.io/projected/feb5d296-405a-4255-812b-8b6f1005b0c3-kube-api-access-jn9fr\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:06.195995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:06.195940    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feb5d296-405a-4255-812b-8b6f1005b0c3-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:06.877740 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:06.877712    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"
Nov 02 12:19:06.877926 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:06.877750    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8" event={"ID":"feb5d296-405a-4255-812b-8b6f1005b0c3","Type":"ContainerDied","Data":"647175c3f9ec84e5bb8b7a2a0f24cf9d94c51c8e7ba0b9a90af0fde0df04ff37"}
Nov 02 12:19:06.877926 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:06.877800    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="647175c3f9ec84e5bb8b7a2a0f24cf9d94c51c8e7ba0b9a90af0fde0df04ff37"
Nov 02 12:19:10.522553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.522518    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"]
Nov 02 12:19:10.523220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.523204    4464 cpu_manager.go:414] "containerMap: removing container" podUID="4acf9653-c380-411c-a0a3-18f49e9f2c9e" containerName="revoke-keys"
Nov 02 12:19:10.523282 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.523222    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4acf9653-c380-411c-a0a3-18f49e9f2c9e" containerName="revoke-keys"
Nov 02 12:19:10.523282 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.523235    4464 cpu_manager.go:414] "containerMap: removing container" podUID="feb5d296-405a-4255-812b-8b6f1005b0c3" containerName="revoke-keys"
Nov 02 12:19:10.523282 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.523240    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb5d296-405a-4255-812b-8b6f1005b0c3" containerName="revoke-keys"
Nov 02 12:19:10.523379 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.523335    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="4acf9653-c380-411c-a0a3-18f49e9f2c9e" containerName="revoke-keys"
Nov 02 12:19:10.523379 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.523344    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb5d296-405a-4255-812b-8b6f1005b0c3" containerName="revoke-keys"
Nov 02 12:19:10.527085 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.527063    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:10.551919 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.551892    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zsql\" (UniqueName: \"kubernetes.io/projected/bcfe0979-1466-4266-aa52-ce7137f909cd-kube-api-access-6zsql\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj\" (UID: \"bcfe0979-1466-4266-aa52-ce7137f909cd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:10.552084 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.552062    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfe0979-1466-4266-aa52-ce7137f909cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj\" (UID: \"bcfe0979-1466-4266-aa52-ce7137f909cd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:10.552151 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.552136    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcfe0979-1466-4266-aa52-ce7137f909cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj\" (UID: \"bcfe0979-1466-4266-aa52-ce7137f909cd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:10.552183 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.552162    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcfe0979-1466-4266-aa52-ce7137f909cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj\" (UID: \"bcfe0979-1466-4266-aa52-ce7137f909cd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:10.584898 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.584869    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"]
Nov 02 12:19:10.630939 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.630905    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"kube-root-ca.crt\""
Nov 02 12:19:10.630939 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.630905    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"openshift-service-ca.crt\""
Nov 02 12:19:10.631154 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.630941    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"routing-overlay\""
Nov 02 12:19:10.631154 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.630933    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-50db61f22b2e\""
Nov 02 12:19:10.631154 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.630943    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-14e0f93131b7\""
Nov 02 12:19:10.631154 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.630919    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3a78b4\""
Nov 02 12:19:10.631154 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.631024    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"payload-processing-external-model-e2e--a8ab13a3-dockercfg-wmcbx\""
Nov 02 12:19:10.653202 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.653173    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfe0979-1466-4266-aa52-ce7137f909cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj\" (UID: \"bcfe0979-1466-4266-aa52-ce7137f909cd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:10.653328 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.653242    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcfe0979-1466-4266-aa52-ce7137f909cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj\" (UID: \"bcfe0979-1466-4266-aa52-ce7137f909cd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:10.653328 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.653273    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcfe0979-1466-4266-aa52-ce7137f909cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj\" (UID: \"bcfe0979-1466-4266-aa52-ce7137f909cd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:10.653483 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.653461    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zsql\" (UniqueName: \"kubernetes.io/projected/bcfe0979-1466-4266-aa52-ce7137f909cd-kube-api-access-6zsql\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj\" (UID: \"bcfe0979-1466-4266-aa52-ce7137f909cd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:10.654088 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.654065    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcfe0979-1466-4266-aa52-ce7137f909cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj\" (UID: \"bcfe0979-1466-4266-aa52-ce7137f909cd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:10.654178 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.654124    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfe0979-1466-4266-aa52-ce7137f909cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj\" (UID: \"bcfe0979-1466-4266-aa52-ce7137f909cd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:10.655875 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.655853    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcfe0979-1466-4266-aa52-ce7137f909cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj\" (UID: \"bcfe0979-1466-4266-aa52-ce7137f909cd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:10.715275 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.715245    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zsql\" (UniqueName: \"kubernetes.io/projected/bcfe0979-1466-4266-aa52-ce7137f909cd-kube-api-access-6zsql\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj\" (UID: \"bcfe0979-1466-4266-aa52-ce7137f909cd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:10.841150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:10.841064    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:11.218564 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:19:11.218534    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfe0979_1466_4266_aa52_ce7137f909cd.slice/crio-929cd9561caf5df78c5a97447da5a160f5041e3e241330cba6abebc52cc955d5 WatchSource:0}: Error finding container 929cd9561caf5df78c5a97447da5a160f5041e3e241330cba6abebc52cc955d5: Status 404 returned error can't find the container with id 929cd9561caf5df78c5a97447da5a160f5041e3e241330cba6abebc52cc955d5
Nov 02 12:19:11.219093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:11.218932    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"]
Nov 02 12:19:11.931451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:11.931389    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj" event={"ID":"bcfe0979-1466-4266-aa52-ce7137f909cd","Type":"ContainerStarted","Data":"25ab589686016372419c17aaee7fd9a3bd7a55451febd88c9cbff891dadb244e"}
Nov 02 12:19:11.931847 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:11.931475    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:11.931847 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:11.931497    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj" event={"ID":"bcfe0979-1466-4266-aa52-ce7137f909cd","Type":"ContainerStarted","Data":"929cd9561caf5df78c5a97447da5a160f5041e3e241330cba6abebc52cc955d5"}
Nov 02 12:19:12.052370 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:12.052310    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj" podStartSLOduration=2.052292608 podStartE2EDuration="2.052292608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:19:12.05049641 +0000 UTC m=+1380.614204820" watchObservedRunningTime="2026-11-02 12:19:12.052292608 +0000 UTC m=+1380.616001018"
Nov 02 12:19:12.060311 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:12.060281    4464 scope.go:122] "RemoveContainer" containerID="fc139f81238554f8b46dadcf71a77af423180febdf21e02bcdc493cb09b0040f"
Nov 02 12:19:12.088330 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:12.088308    4464 scope.go:122] "RemoveContainer" containerID="41b74a0a03f7fc04b6bac1240f2bf35b998ee5845ebe3cc0d75a31b53b4fd5f2"
Nov 02 12:19:12.097920 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:12.097902    4464 scope.go:122] "RemoveContainer" containerID="38d832646261daa3aa56688202e407311bf9e8ffec9ee7d2ab435acbd659795e"
Nov 02 12:19:13.066590 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:13.066554    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:19:15.499041 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:15.499004    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"]
Nov 02 12:19:15.506058 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:15.506028    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"
Nov 02 12:19:15.547855 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:15.547823    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj5zn\" (UniqueName: \"kubernetes.io/projected/ec5fbab1-5776-4d45-8beb-19c36dd6467f-kube-api-access-rj5zn\") pod \"maas-api-revoke-sub-8d9c1ba7-nlpgb\" (UID: \"ec5fbab1-5776-4d45-8beb-19c36dd6467f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"
Nov 02 12:19:15.548029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:15.547988    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5fbab1-5776-4d45-8beb-19c36dd6467f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d9c1ba7-nlpgb\" (UID: \"ec5fbab1-5776-4d45-8beb-19c36dd6467f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"
Nov 02 12:19:15.548029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:15.548025    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5fbab1-5776-4d45-8beb-19c36dd6467f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d9c1ba7-nlpgb\" (UID: \"ec5fbab1-5776-4d45-8beb-19c36dd6467f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"
Nov 02 12:19:15.557960 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:15.557930    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"]
Nov 02 12:19:15.559774 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:15.559752    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:19:15.560014 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:15.559938    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:19:15.600575 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:15.600543    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:19:15.600575 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:15.600543    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:19:15.649122 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:15.649092    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5fbab1-5776-4d45-8beb-19c36dd6467f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d9c1ba7-nlpgb\" (UID: \"ec5fbab1-5776-4d45-8beb-19c36dd6467f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"
Nov 02 12:19:15.649308 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:15.649148    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5fbab1-5776-4d45-8beb-19c36dd6467f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d9c1ba7-nlpgb\" (UID: \"ec5fbab1-5776-4d45-8beb-19c36dd6467f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"
Nov 02 12:19:15.649308 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:15.649210    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj5zn\" (UniqueName: \"kubernetes.io/projected/ec5fbab1-5776-4d45-8beb-19c36dd6467f-kube-api-access-rj5zn\") pod \"maas-api-revoke-sub-8d9c1ba7-nlpgb\" (UID: \"ec5fbab1-5776-4d45-8beb-19c36dd6467f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"
Nov 02 12:19:15.649785 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:15.649761    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5fbab1-5776-4d45-8beb-19c36dd6467f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d9c1ba7-nlpgb\" (UID: \"ec5fbab1-5776-4d45-8beb-19c36dd6467f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"
Nov 02 12:19:15.651707 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:15.651686    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5fbab1-5776-4d45-8beb-19c36dd6467f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d9c1ba7-nlpgb\" (UID: \"ec5fbab1-5776-4d45-8beb-19c36dd6467f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"
Nov 02 12:19:15.711638 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:15.711610    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj5zn\" (UniqueName: \"kubernetes.io/projected/ec5fbab1-5776-4d45-8beb-19c36dd6467f-kube-api-access-rj5zn\") pod \"maas-api-revoke-sub-8d9c1ba7-nlpgb\" (UID: \"ec5fbab1-5776-4d45-8beb-19c36dd6467f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"
Nov 02 12:19:15.822451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:15.822378    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"
Nov 02 12:19:16.204502 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:19:16.204469    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5fbab1_5776_4d45_8beb_19c36dd6467f.slice/crio-00689ef1c73f51891a32eb5f46148655bd627dab7cec707f98219afe40cde615 WatchSource:0}: Error finding container 00689ef1c73f51891a32eb5f46148655bd627dab7cec707f98219afe40cde615: Status 404 returned error can't find the container with id 00689ef1c73f51891a32eb5f46148655bd627dab7cec707f98219afe40cde615
Nov 02 12:19:16.205167 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:16.205142    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"]
Nov 02 12:19:16.971094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:16.971062    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb" event={"ID":"ec5fbab1-5776-4d45-8beb-19c36dd6467f","Type":"ContainerStarted","Data":"b689441e33d3512ccfe36097ea8963aad7520b1e7d7f9b608e7a082814ac2a5e"}
Nov 02 12:19:16.971617 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:16.971115    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb" event={"ID":"ec5fbab1-5776-4d45-8beb-19c36dd6467f","Type":"ContainerStarted","Data":"00689ef1c73f51891a32eb5f46148655bd627dab7cec707f98219afe40cde615"}
Nov 02 12:19:17.095230 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:17.095177    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb" podStartSLOduration=2.095163798 podStartE2EDuration="2.095163798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:19:17.094363667 +0000 UTC m=+1385.658072095" watchObservedRunningTime="2026-11-02 12:19:17.095163798 +0000 UTC m=+1385.658872206"
Nov 02 12:19:17.979699 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:17.979663    4464 generic.go:356] "Generic (PLEG): container finished" podID="ec5fbab1-5776-4d45-8beb-19c36dd6467f" containerID="b689441e33d3512ccfe36097ea8963aad7520b1e7d7f9b608e7a082814ac2a5e" exitCode=0
Nov 02 12:19:17.980089 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:17.979752    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb" event={"ID":"ec5fbab1-5776-4d45-8beb-19c36dd6467f","Type":"ContainerDied","Data":"b689441e33d3512ccfe36097ea8963aad7520b1e7d7f9b608e7a082814ac2a5e"}
Nov 02 12:19:18.249164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:18.249079    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"]
Nov 02 12:19:18.253633 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:18.253611    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"
Nov 02 12:19:18.275118 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:18.275095    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r29vk\" (UniqueName: \"kubernetes.io/projected/3af88032-0c31-4bb0-a7a9-032f74417e69-kube-api-access-r29vk\") pod \"maas-api-revoke-sub-d755c31c-4jg9p\" (UID: \"3af88032-0c31-4bb0-a7a9-032f74417e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"
Nov 02 12:19:18.275226 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:18.275141    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3af88032-0c31-4bb0-a7a9-032f74417e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d755c31c-4jg9p\" (UID: \"3af88032-0c31-4bb0-a7a9-032f74417e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"
Nov 02 12:19:18.275326 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:18.275305    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3af88032-0c31-4bb0-a7a9-032f74417e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d755c31c-4jg9p\" (UID: \"3af88032-0c31-4bb0-a7a9-032f74417e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"
Nov 02 12:19:18.311768 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:18.311741    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"]
Nov 02 12:19:18.376255 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:18.376225    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r29vk\" (UniqueName: \"kubernetes.io/projected/3af88032-0c31-4bb0-a7a9-032f74417e69-kube-api-access-r29vk\") pod \"maas-api-revoke-sub-d755c31c-4jg9p\" (UID: \"3af88032-0c31-4bb0-a7a9-032f74417e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"
Nov 02 12:19:18.376447 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:18.376293    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3af88032-0c31-4bb0-a7a9-032f74417e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d755c31c-4jg9p\" (UID: \"3af88032-0c31-4bb0-a7a9-032f74417e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"
Nov 02 12:19:18.376522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:18.376452    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3af88032-0c31-4bb0-a7a9-032f74417e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d755c31c-4jg9p\" (UID: \"3af88032-0c31-4bb0-a7a9-032f74417e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"
Nov 02 12:19:18.377085 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:18.377048    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3af88032-0c31-4bb0-a7a9-032f74417e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d755c31c-4jg9p\" (UID: \"3af88032-0c31-4bb0-a7a9-032f74417e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"
Nov 02 12:19:18.378508 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:18.378490    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3af88032-0c31-4bb0-a7a9-032f74417e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d755c31c-4jg9p\" (UID: \"3af88032-0c31-4bb0-a7a9-032f74417e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"
Nov 02 12:19:18.438267 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:18.438236    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r29vk\" (UniqueName: \"kubernetes.io/projected/3af88032-0c31-4bb0-a7a9-032f74417e69-kube-api-access-r29vk\") pod \"maas-api-revoke-sub-d755c31c-4jg9p\" (UID: \"3af88032-0c31-4bb0-a7a9-032f74417e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"
Nov 02 12:19:18.567271 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:18.567194    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"
Nov 02 12:19:18.953111 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:19:18.953083    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af88032_0c31_4bb0_a7a9_032f74417e69.slice/crio-fe7075ae8fc4a4a69aa79ad346893ea685885e135cec47fa11435c799ea02035 WatchSource:0}: Error finding container fe7075ae8fc4a4a69aa79ad346893ea685885e135cec47fa11435c799ea02035: Status 404 returned error can't find the container with id fe7075ae8fc4a4a69aa79ad346893ea685885e135cec47fa11435c799ea02035
Nov 02 12:19:18.954874 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:18.954846    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"]
Nov 02 12:19:18.988510 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:18.988482    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p" event={"ID":"3af88032-0c31-4bb0-a7a9-032f74417e69","Type":"ContainerStarted","Data":"fe7075ae8fc4a4a69aa79ad346893ea685885e135cec47fa11435c799ea02035"}
Nov 02 12:19:19.056188 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:19.056158    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"]
Nov 02 12:19:19.064197 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:19.064163    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-156b3cc1-pkvvg"]
Nov 02 12:19:19.170411 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:19.170386    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"
Nov 02 12:19:19.190388 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:19.190357    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec5fbab1-5776-4d45-8beb-19c36dd6467f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5fbab1-5776-4d45-8beb-19c36dd6467f-maas-api-openshift-ca\") pod \"ec5fbab1-5776-4d45-8beb-19c36dd6467f\" (UID: \"ec5fbab1-5776-4d45-8beb-19c36dd6467f\") "
Nov 02 12:19:19.190519 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:19.190393    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec5fbab1-5776-4d45-8beb-19c36dd6467f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5fbab1-5776-4d45-8beb-19c36dd6467f-maas-api-xks-ca\") pod \"ec5fbab1-5776-4d45-8beb-19c36dd6467f\" (UID: \"ec5fbab1-5776-4d45-8beb-19c36dd6467f\") "
Nov 02 12:19:19.190519 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:19.190481    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec5fbab1-5776-4d45-8beb-19c36dd6467f-kube-api-access-rj5zn\" (UniqueName: \"kubernetes.io/projected/ec5fbab1-5776-4d45-8beb-19c36dd6467f-kube-api-access-rj5zn\") pod \"ec5fbab1-5776-4d45-8beb-19c36dd6467f\" (UID: \"ec5fbab1-5776-4d45-8beb-19c36dd6467f\") "
Nov 02 12:19:19.190692 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:19.190670    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec5fbab1-5776-4d45-8beb-19c36dd6467f-maas-api-openshift-ca" pod "ec5fbab1-5776-4d45-8beb-19c36dd6467f" (UID: "ec5fbab1-5776-4d45-8beb-19c36dd6467f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:19:19.191009 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:19.190987    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec5fbab1-5776-4d45-8beb-19c36dd6467f-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:19.192303 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:19.192281    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec5fbab1-5776-4d45-8beb-19c36dd6467f-maas-api-xks-ca" pod "ec5fbab1-5776-4d45-8beb-19c36dd6467f" (UID: "ec5fbab1-5776-4d45-8beb-19c36dd6467f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:19:19.192420 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:19.192401    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5fbab1-5776-4d45-8beb-19c36dd6467f-kube-api-access-rj5zn" pod "ec5fbab1-5776-4d45-8beb-19c36dd6467f" (UID: "ec5fbab1-5776-4d45-8beb-19c36dd6467f"). InnerVolumeSpecName "kube-api-access-rj5zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:19:19.292752 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:19.292720    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec5fbab1-5776-4d45-8beb-19c36dd6467f-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:19.292752 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:19.292754    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj5zn\" (UniqueName: \"kubernetes.io/projected/ec5fbab1-5776-4d45-8beb-19c36dd6467f-kube-api-access-rj5zn\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:19.583599 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:19.583521    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f3ea0f-f725-4620-93ab-47fc867878ce" path="/var/lib/kubelet/pods/74f3ea0f-f725-4620-93ab-47fc867878ce/volumes"
Nov 02 12:19:19.997660 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:19.997622    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p" event={"ID":"3af88032-0c31-4bb0-a7a9-032f74417e69","Type":"ContainerStarted","Data":"20319900b15ec34ecb65bbe8d753418726ed0454a411ba38f722b21fea5d0a22"}
Nov 02 12:19:19.999167 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:19.999138    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"
Nov 02 12:19:19.999289 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:19.999160    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb" event={"ID":"ec5fbab1-5776-4d45-8beb-19c36dd6467f","Type":"ContainerDied","Data":"00689ef1c73f51891a32eb5f46148655bd627dab7cec707f98219afe40cde615"}
Nov 02 12:19:19.999289 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:19.999202    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00689ef1c73f51891a32eb5f46148655bd627dab7cec707f98219afe40cde615"
Nov 02 12:19:20.129528 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:20.129476    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p" podStartSLOduration=2.129462038 podStartE2EDuration="2.129462038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:19:20.125479283 +0000 UTC m=+1388.689187704" watchObservedRunningTime="2026-11-02 12:19:20.129462038 +0000 UTC m=+1388.693170484"
Nov 02 12:19:21.008390 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:21.008350    4464 generic.go:356] "Generic (PLEG): container finished" podID="3af88032-0c31-4bb0-a7a9-032f74417e69" containerID="20319900b15ec34ecb65bbe8d753418726ed0454a411ba38f722b21fea5d0a22" exitCode=0
Nov 02 12:19:21.008787 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:21.008396    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p" event={"ID":"3af88032-0c31-4bb0-a7a9-032f74417e69","Type":"ContainerDied","Data":"20319900b15ec34ecb65bbe8d753418726ed0454a411ba38f722b21fea5d0a22"}
Nov 02 12:19:21.611854 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:21.611820    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"]
Nov 02 12:19:21.612176 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:21.612142    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd" podUID="e127e191-c68b-4d5f-8c6c-d6b42598e2b6" containerName="istio-proxy" containerID="cri-o://fc1515a341b1ff33b56ebb6021e16046fc87894a2bca733765ca6d0d0c167721" gracePeriod=30
Nov 02 12:19:22.162803 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:22.162779    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"
Nov 02 12:19:22.235736 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:22.235700    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3af88032-0c31-4bb0-a7a9-032f74417e69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3af88032-0c31-4bb0-a7a9-032f74417e69-maas-api-xks-ca\") pod \"3af88032-0c31-4bb0-a7a9-032f74417e69\" (UID: \"3af88032-0c31-4bb0-a7a9-032f74417e69\") "
Nov 02 12:19:22.235913 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:22.235848    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3af88032-0c31-4bb0-a7a9-032f74417e69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3af88032-0c31-4bb0-a7a9-032f74417e69-maas-api-openshift-ca\") pod \"3af88032-0c31-4bb0-a7a9-032f74417e69\" (UID: \"3af88032-0c31-4bb0-a7a9-032f74417e69\") "
Nov 02 12:19:22.236009 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:22.235941    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3af88032-0c31-4bb0-a7a9-032f74417e69-kube-api-access-r29vk\" (UniqueName: \"kubernetes.io/projected/3af88032-0c31-4bb0-a7a9-032f74417e69-kube-api-access-r29vk\") pod \"3af88032-0c31-4bb0-a7a9-032f74417e69\" (UID: \"3af88032-0c31-4bb0-a7a9-032f74417e69\") "
Nov 02 12:19:22.236214 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:22.236188    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af88032-0c31-4bb0-a7a9-032f74417e69-maas-api-openshift-ca" pod "3af88032-0c31-4bb0-a7a9-032f74417e69" (UID: "3af88032-0c31-4bb0-a7a9-032f74417e69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:19:22.236488 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:22.236470    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3af88032-0c31-4bb0-a7a9-032f74417e69-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:22.237835 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:22.237810    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3af88032-0c31-4bb0-a7a9-032f74417e69-maas-api-xks-ca" pod "3af88032-0c31-4bb0-a7a9-032f74417e69" (UID: "3af88032-0c31-4bb0-a7a9-032f74417e69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:19:22.237934 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:22.237889    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af88032-0c31-4bb0-a7a9-032f74417e69-kube-api-access-r29vk" pod "3af88032-0c31-4bb0-a7a9-032f74417e69" (UID: "3af88032-0c31-4bb0-a7a9-032f74417e69"). InnerVolumeSpecName "kube-api-access-r29vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:19:22.338261 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:22.338175    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r29vk\" (UniqueName: \"kubernetes.io/projected/3af88032-0c31-4bb0-a7a9-032f74417e69-kube-api-access-r29vk\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:22.338261 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:22.338213    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3af88032-0c31-4bb0-a7a9-032f74417e69-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:22.943107 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:22.943075    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:23.024320 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:23.024290    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"
Nov 02 12:19:23.024320 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:23.024311    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p" event={"ID":"3af88032-0c31-4bb0-a7a9-032f74417e69","Type":"ContainerDied","Data":"fe7075ae8fc4a4a69aa79ad346893ea685885e135cec47fa11435c799ea02035"}
Nov 02 12:19:23.024530 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:23.024358    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe7075ae8fc4a4a69aa79ad346893ea685885e135cec47fa11435c799ea02035"
Nov 02 12:19:26.882016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.881942    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:19:26.995248 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.995216    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-kube-api-access-gplw2\" (UniqueName: \"kubernetes.io/projected/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-kube-api-access-gplw2\") pod \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") "
Nov 02 12:19:26.995432 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.995254    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-token\" (UniqueName: \"kubernetes.io/projected/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-token\") pod \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") "
Nov 02 12:19:26.995432 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.995293    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-data\") pod \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") "
Nov 02 12:19:26.995432 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.995341    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istiod-ca-cert\") pod \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") "
Nov 02 12:19:26.995432 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.995375    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-envoy\") pod \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") "
Nov 02 12:19:26.995432 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.995429    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-workload-certs\") pod \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") "
Nov 02 12:19:26.995651 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.995554    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-credential-socket\") pod \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") "
Nov 02 12:19:26.995651 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.995587    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-workload-socket\") pod \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") "
Nov 02 12:19:26.995756 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.995653    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-data" pod "e127e191-c68b-4d5f-8c6c-d6b42598e2b6" (UID: "e127e191-c68b-4d5f-8c6c-d6b42598e2b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:19:26.995756 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.995703    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-podinfo\") pod \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\" (UID: \"e127e191-c68b-4d5f-8c6c-d6b42598e2b6\") "
Nov 02 12:19:26.995868 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.995785    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istiod-ca-cert" pod "e127e191-c68b-4d5f-8c6c-d6b42598e2b6" (UID: "e127e191-c68b-4d5f-8c6c-d6b42598e2b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:19:26.995941 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.995890    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-credential-socket" pod "e127e191-c68b-4d5f-8c6c-d6b42598e2b6" (UID: "e127e191-c68b-4d5f-8c6c-d6b42598e2b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:19:26.995941 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.995920    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-workload-socket" pod "e127e191-c68b-4d5f-8c6c-d6b42598e2b6" (UID: "e127e191-c68b-4d5f-8c6c-d6b42598e2b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:19:26.996376 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.996051    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-workload-certs" pod "e127e191-c68b-4d5f-8c6c-d6b42598e2b6" (UID: "e127e191-c68b-4d5f-8c6c-d6b42598e2b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:19:26.996376 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.996261    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:26.996376 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.996284    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:26.996376 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.996301    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:26.996376 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.996315    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:26.996376 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.996329    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:26.997544 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.997517    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-kube-api-access-gplw2" pod "e127e191-c68b-4d5f-8c6c-d6b42598e2b6" (UID: "e127e191-c68b-4d5f-8c6c-d6b42598e2b6"). InnerVolumeSpecName "kube-api-access-gplw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:19:26.997691 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.997669    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-envoy" pod "e127e191-c68b-4d5f-8c6c-d6b42598e2b6" (UID: "e127e191-c68b-4d5f-8c6c-d6b42598e2b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:19:26.997784 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.997768    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-podinfo" pod "e127e191-c68b-4d5f-8c6c-d6b42598e2b6" (UID: "e127e191-c68b-4d5f-8c6c-d6b42598e2b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:19:26.997964 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:26.997946    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-token" pod "e127e191-c68b-4d5f-8c6c-d6b42598e2b6" (UID: "e127e191-c68b-4d5f-8c6c-d6b42598e2b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:19:27.055313 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.055280    4464 generic.go:356] "Generic (PLEG): container finished" podID="e127e191-c68b-4d5f-8c6c-d6b42598e2b6" containerID="fc1515a341b1ff33b56ebb6021e16046fc87894a2bca733765ca6d0d0c167721" exitCode=0
Nov 02 12:19:27.055493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.055336    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"
Nov 02 12:19:27.055493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.055384    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd" event={"ID":"e127e191-c68b-4d5f-8c6c-d6b42598e2b6","Type":"ContainerDied","Data":"fc1515a341b1ff33b56ebb6021e16046fc87894a2bca733765ca6d0d0c167721"}
Nov 02 12:19:27.055493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.055459    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd" event={"ID":"e127e191-c68b-4d5f-8c6c-d6b42598e2b6","Type":"ContainerDied","Data":"5ba48c14181ff116b155596d79797253adb78d5e5ce1c57af48c88bf22efa11e"}
Nov 02 12:19:27.055493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.055482    4464 scope.go:122] "RemoveContainer" containerID="fc1515a341b1ff33b56ebb6021e16046fc87894a2bca733765ca6d0d0c167721"
Nov 02 12:19:27.074361 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.074339    4464 scope.go:122] "RemoveContainer" containerID="ddb98e1aa1f928a20c72e4980aa69c9eb00050ecaa7c980237aba499d493ecbf"
Nov 02 12:19:27.086727 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.086700    4464 scope.go:122] "RemoveContainer" containerID="fc1515a341b1ff33b56ebb6021e16046fc87894a2bca733765ca6d0d0c167721"
Nov 02 12:19:27.087045 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:19:27.087025    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1515a341b1ff33b56ebb6021e16046fc87894a2bca733765ca6d0d0c167721\": container with ID starting with fc1515a341b1ff33b56ebb6021e16046fc87894a2bca733765ca6d0d0c167721 not found: ID does not exist" containerID="fc1515a341b1ff33b56ebb6021e16046fc87894a2bca733765ca6d0d0c167721"
Nov 02 12:19:27.087137 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.087053    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1515a341b1ff33b56ebb6021e16046fc87894a2bca733765ca6d0d0c167721"} err="failed to get container status \"fc1515a341b1ff33b56ebb6021e16046fc87894a2bca733765ca6d0d0c167721\": rpc error: code = NotFound desc = could not find container \"fc1515a341b1ff33b56ebb6021e16046fc87894a2bca733765ca6d0d0c167721\": container with ID starting with fc1515a341b1ff33b56ebb6021e16046fc87894a2bca733765ca6d0d0c167721 not found: ID does not exist"
Nov 02 12:19:27.087137 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.087073    4464 scope.go:122] "RemoveContainer" containerID="ddb98e1aa1f928a20c72e4980aa69c9eb00050ecaa7c980237aba499d493ecbf"
Nov 02 12:19:27.087326 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:19:27.087306    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb98e1aa1f928a20c72e4980aa69c9eb00050ecaa7c980237aba499d493ecbf\": container with ID starting with ddb98e1aa1f928a20c72e4980aa69c9eb00050ecaa7c980237aba499d493ecbf not found: ID does not exist" containerID="ddb98e1aa1f928a20c72e4980aa69c9eb00050ecaa7c980237aba499d493ecbf"
Nov 02 12:19:27.087367 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.087332    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb98e1aa1f928a20c72e4980aa69c9eb00050ecaa7c980237aba499d493ecbf"} err="failed to get container status \"ddb98e1aa1f928a20c72e4980aa69c9eb00050ecaa7c980237aba499d493ecbf\": rpc error: code = NotFound desc = could not find container \"ddb98e1aa1f928a20c72e4980aa69c9eb00050ecaa7c980237aba499d493ecbf\": container with ID starting with ddb98e1aa1f928a20c72e4980aa69c9eb00050ecaa7c980237aba499d493ecbf not found: ID does not exist"
Nov 02 12:19:27.097062 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.097034    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:27.097062 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.097057    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gplw2\" (UniqueName: \"kubernetes.io/projected/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-kube-api-access-gplw2\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:27.097062 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.097067    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:27.097281 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.097075    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e127e191-c68b-4d5f-8c6c-d6b42598e2b6-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:27.239317 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.239280    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"]
Nov 02 12:19:27.246440 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.246416    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7bd72e-openshift-default-6fd8c7d56f-ld8bd"]
Nov 02 12:19:27.519438 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.519356    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"]
Nov 02 12:19:27.582446 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.582416    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e127e191-c68b-4d5f-8c6c-d6b42598e2b6" path="/var/lib/kubelet/pods/e127e191-c68b-4d5f-8c6c-d6b42598e2b6/volumes"
Nov 02 12:19:27.641941 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:27.641906    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"]
Nov 02 12:19:28.291882 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:19:28.291850    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:19:29.165995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.163363    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"]
Nov 02 12:19:29.165995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.165521    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e127e191-c68b-4d5f-8c6c-d6b42598e2b6" containerName="istio-proxy"
Nov 02 12:19:29.165995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.165560    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e127e191-c68b-4d5f-8c6c-d6b42598e2b6" containerName="istio-proxy"
Nov 02 12:19:29.165995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.165585    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e127e191-c68b-4d5f-8c6c-d6b42598e2b6" containerName="istio-proxy"
Nov 02 12:19:29.165995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.165594    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e127e191-c68b-4d5f-8c6c-d6b42598e2b6" containerName="istio-proxy"
Nov 02 12:19:29.165995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.165604    4464 cpu_manager.go:414] "containerMap: removing container" podUID="ec5fbab1-5776-4d45-8beb-19c36dd6467f" containerName="revoke-keys"
Nov 02 12:19:29.165995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.165611    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec5fbab1-5776-4d45-8beb-19c36dd6467f" containerName="revoke-keys"
Nov 02 12:19:29.165995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.165643    4464 cpu_manager.go:414] "containerMap: removing container" podUID="3af88032-0c31-4bb0-a7a9-032f74417e69" containerName="revoke-keys"
Nov 02 12:19:29.165995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.165651    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3af88032-0c31-4bb0-a7a9-032f74417e69" containerName="revoke-keys"
Nov 02 12:19:29.165995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.165885    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e127e191-c68b-4d5f-8c6c-d6b42598e2b6" containerName="istio-proxy"
Nov 02 12:19:29.165995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.165901    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec5fbab1-5776-4d45-8beb-19c36dd6467f" containerName="revoke-keys"
Nov 02 12:19:29.165995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.165910    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="3af88032-0c31-4bb0-a7a9-032f74417e69" containerName="revoke-keys"
Nov 02 12:19:29.173010 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.172947    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"]
Nov 02 12:19:29.173153 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.173138    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"
Nov 02 12:19:29.228453 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.228419    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:19:29.228453 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.228420    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:19:29.228653 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.228427    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:19:29.228653 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.228478    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:19:29.376998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.374887    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f14d4e2-8f92-4a32-885e-5a132b131e72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b641c6c5-vbnw4\" (UID: \"6f14d4e2-8f92-4a32-885e-5a132b131e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"
Nov 02 12:19:29.376998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.375073    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f14d4e2-8f92-4a32-885e-5a132b131e72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b641c6c5-vbnw4\" (UID: \"6f14d4e2-8f92-4a32-885e-5a132b131e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"
Nov 02 12:19:29.376998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.375191    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rgxl\" (UniqueName: \"kubernetes.io/projected/6f14d4e2-8f92-4a32-885e-5a132b131e72-kube-api-access-7rgxl\") pod \"maas-api-revoke-sub-b641c6c5-vbnw4\" (UID: \"6f14d4e2-8f92-4a32-885e-5a132b131e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"
Nov 02 12:19:29.491999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.484290    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f14d4e2-8f92-4a32-885e-5a132b131e72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b641c6c5-vbnw4\" (UID: \"6f14d4e2-8f92-4a32-885e-5a132b131e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"
Nov 02 12:19:29.491999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.484392    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f14d4e2-8f92-4a32-885e-5a132b131e72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b641c6c5-vbnw4\" (UID: \"6f14d4e2-8f92-4a32-885e-5a132b131e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"
Nov 02 12:19:29.491999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.484470    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rgxl\" (UniqueName: \"kubernetes.io/projected/6f14d4e2-8f92-4a32-885e-5a132b131e72-kube-api-access-7rgxl\") pod \"maas-api-revoke-sub-b641c6c5-vbnw4\" (UID: \"6f14d4e2-8f92-4a32-885e-5a132b131e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"
Nov 02 12:19:29.491999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.486101    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f14d4e2-8f92-4a32-885e-5a132b131e72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b641c6c5-vbnw4\" (UID: \"6f14d4e2-8f92-4a32-885e-5a132b131e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"
Nov 02 12:19:29.518009 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.515731    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f14d4e2-8f92-4a32-885e-5a132b131e72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b641c6c5-vbnw4\" (UID: \"6f14d4e2-8f92-4a32-885e-5a132b131e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"
Nov 02 12:19:29.546859 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.546823    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rgxl\" (UniqueName: \"kubernetes.io/projected/6f14d4e2-8f92-4a32-885e-5a132b131e72-kube-api-access-7rgxl\") pod \"maas-api-revoke-sub-b641c6c5-vbnw4\" (UID: \"6f14d4e2-8f92-4a32-885e-5a132b131e72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"
Nov 02 12:19:29.794376 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:29.794302    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"
Nov 02 12:19:30.214055 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:30.209091    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"]
Nov 02 12:19:31.218405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:31.218367    4464 generic.go:356] "Generic (PLEG): container finished" podID="6f14d4e2-8f92-4a32-885e-5a132b131e72" containerID="347f04988127f4fcc7ec63da51fbaa60920ae555bce1064fe4d1d4725784b194" exitCode=0
Nov 02 12:19:31.218862 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:31.218513    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4" event={"ID":"6f14d4e2-8f92-4a32-885e-5a132b131e72","Type":"ContainerDied","Data":"347f04988127f4fcc7ec63da51fbaa60920ae555bce1064fe4d1d4725784b194"}
Nov 02 12:19:31.218862 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:31.218561    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4" event={"ID":"6f14d4e2-8f92-4a32-885e-5a132b131e72","Type":"ContainerStarted","Data":"c75cf8d65b8b85a1e2e7e24f67df01f67776c9c3db0fb2345a56a565f6aef550"}
Nov 02 12:19:32.520185 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:32.520096    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29" podUID="6ef3a987-e107-4351-9e85-cafeecf0a708" containerName="payload-processing" containerID="cri-o://b430a2037a7be05b52705adcb988d5fbc414c4f65a6fe50aaf5f08b610f46dbf" gracePeriod=25
Nov 02 12:19:32.544164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:32.544137    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"
Nov 02 12:19:32.639059 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:32.638876    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f14d4e2-8f92-4a32-885e-5a132b131e72-kube-api-access-7rgxl\" (UniqueName: \"kubernetes.io/projected/6f14d4e2-8f92-4a32-885e-5a132b131e72-kube-api-access-7rgxl\") pod \"6f14d4e2-8f92-4a32-885e-5a132b131e72\" (UID: \"6f14d4e2-8f92-4a32-885e-5a132b131e72\") "
Nov 02 12:19:32.639059 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:32.639049    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f14d4e2-8f92-4a32-885e-5a132b131e72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f14d4e2-8f92-4a32-885e-5a132b131e72-maas-api-openshift-ca\") pod \"6f14d4e2-8f92-4a32-885e-5a132b131e72\" (UID: \"6f14d4e2-8f92-4a32-885e-5a132b131e72\") "
Nov 02 12:19:32.639329 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:32.639090    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f14d4e2-8f92-4a32-885e-5a132b131e72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f14d4e2-8f92-4a32-885e-5a132b131e72-maas-api-xks-ca\") pod \"6f14d4e2-8f92-4a32-885e-5a132b131e72\" (UID: \"6f14d4e2-8f92-4a32-885e-5a132b131e72\") "
Nov 02 12:19:32.640074 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:32.640039    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f14d4e2-8f92-4a32-885e-5a132b131e72-maas-api-openshift-ca" pod "6f14d4e2-8f92-4a32-885e-5a132b131e72" (UID: "6f14d4e2-8f92-4a32-885e-5a132b131e72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:19:32.642085 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:32.642054    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f14d4e2-8f92-4a32-885e-5a132b131e72-kube-api-access-7rgxl" pod "6f14d4e2-8f92-4a32-885e-5a132b131e72" (UID: "6f14d4e2-8f92-4a32-885e-5a132b131e72"). InnerVolumeSpecName "kube-api-access-7rgxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:19:32.643699 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:32.643675    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v" podUID="01b8183e-899a-42ca-89b1-9a9db37b239a" containerName="payload-pre-processing" containerID="cri-o://5265125648a116c355e29a8b6dd0975f34974a855a2acec498b8c2aaa8292a45" gracePeriod=25
Nov 02 12:19:32.643798 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:32.643710    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f14d4e2-8f92-4a32-885e-5a132b131e72-maas-api-xks-ca" pod "6f14d4e2-8f92-4a32-885e-5a132b131e72" (UID: "6f14d4e2-8f92-4a32-885e-5a132b131e72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:19:32.749214 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:32.742029    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rgxl\" (UniqueName: \"kubernetes.io/projected/6f14d4e2-8f92-4a32-885e-5a132b131e72-kube-api-access-7rgxl\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:32.749214 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:32.742071    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f14d4e2-8f92-4a32-885e-5a132b131e72-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:32.749214 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:32.742090    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f14d4e2-8f92-4a32-885e-5a132b131e72-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:33.242432 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:33.242395    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"
Nov 02 12:19:33.242636 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:33.242430    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4" event={"ID":"6f14d4e2-8f92-4a32-885e-5a132b131e72","Type":"ContainerDied","Data":"c75cf8d65b8b85a1e2e7e24f67df01f67776c9c3db0fb2345a56a565f6aef550"}
Nov 02 12:19:33.242636 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:33.242477    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c75cf8d65b8b85a1e2e7e24f67df01f67776c9c3db0fb2345a56a565f6aef550"
Nov 02 12:19:33.475705 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:33.475225    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:19:33.475705 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:33.475294    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:19:36.801096 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:36.801057    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29" podUID="6ef3a987-e107-4351-9e85-cafeecf0a708" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:19:36.801565 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:36.801070    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v" podUID="01b8183e-899a-42ca-89b1-9a9db37b239a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:19:36.857870 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:36.857826    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"]
Nov 02 12:19:36.858538 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:36.858513    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6f14d4e2-8f92-4a32-885e-5a132b131e72" containerName="revoke-keys"
Nov 02 12:19:36.858538 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:36.858536    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f14d4e2-8f92-4a32-885e-5a132b131e72" containerName="revoke-keys"
Nov 02 12:19:36.858705 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:36.858663    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f14d4e2-8f92-4a32-885e-5a132b131e72" containerName="revoke-keys"
Nov 02 12:19:36.858705 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:36.858679    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e127e191-c68b-4d5f-8c6c-d6b42598e2b6" containerName="istio-proxy"
Nov 02 12:19:36.862326 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:36.862310    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"
Nov 02 12:19:36.899473 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:36.899448    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84lns\" (UniqueName: \"kubernetes.io/projected/ad2bbef1-1336-4164-a083-c15cf501bfef-kube-api-access-84lns\") pod \"maas-api-revoke-sub-0c9536ab-sgclr\" (UID: \"ad2bbef1-1336-4164-a083-c15cf501bfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"
Nov 02 12:19:36.899571 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:36.899491    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2bbef1-1336-4164-a083-c15cf501bfef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c9536ab-sgclr\" (UID: \"ad2bbef1-1336-4164-a083-c15cf501bfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"
Nov 02 12:19:36.899571 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:36.899560    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2bbef1-1336-4164-a083-c15cf501bfef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c9536ab-sgclr\" (UID: \"ad2bbef1-1336-4164-a083-c15cf501bfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"
Nov 02 12:19:36.919817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:36.919792    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"]
Nov 02 12:19:36.964631 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:36.964600    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:19:36.964767 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:36.964615    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:19:36.964767 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:36.964647    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:19:36.965066 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:36.965050    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:19:37.000439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:37.000416    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2bbef1-1336-4164-a083-c15cf501bfef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c9536ab-sgclr\" (UID: \"ad2bbef1-1336-4164-a083-c15cf501bfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"
Nov 02 12:19:37.000560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:37.000465    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2bbef1-1336-4164-a083-c15cf501bfef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c9536ab-sgclr\" (UID: \"ad2bbef1-1336-4164-a083-c15cf501bfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"
Nov 02 12:19:37.000697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:37.000681    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84lns\" (UniqueName: \"kubernetes.io/projected/ad2bbef1-1336-4164-a083-c15cf501bfef-kube-api-access-84lns\") pod \"maas-api-revoke-sub-0c9536ab-sgclr\" (UID: \"ad2bbef1-1336-4164-a083-c15cf501bfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"
Nov 02 12:19:37.001150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:37.001130    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2bbef1-1336-4164-a083-c15cf501bfef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c9536ab-sgclr\" (UID: \"ad2bbef1-1336-4164-a083-c15cf501bfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"
Nov 02 12:19:37.002865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:37.002843    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2bbef1-1336-4164-a083-c15cf501bfef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c9536ab-sgclr\" (UID: \"ad2bbef1-1336-4164-a083-c15cf501bfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"
Nov 02 12:19:37.090051 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:37.088797    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84lns\" (UniqueName: \"kubernetes.io/projected/ad2bbef1-1336-4164-a083-c15cf501bfef-kube-api-access-84lns\") pod \"maas-api-revoke-sub-0c9536ab-sgclr\" (UID: \"ad2bbef1-1336-4164-a083-c15cf501bfef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"
Nov 02 12:19:37.106680 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:37.105045    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"]
Nov 02 12:19:37.134075 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:37.134041    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-156e3027-nml6k"]
Nov 02 12:19:37.176433 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:37.176404    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"
Nov 02 12:19:37.579624 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:19:37.578961    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2bbef1_1336_4164_a083_c15cf501bfef.slice/crio-4b8e7e2aeeba5793b4e98c72d9d66ca0e42792dd7884642c4dd71325c50409b5 WatchSource:0}: Error finding container 4b8e7e2aeeba5793b4e98c72d9d66ca0e42792dd7884642c4dd71325c50409b5: Status 404 returned error can't find the container with id 4b8e7e2aeeba5793b4e98c72d9d66ca0e42792dd7884642c4dd71325c50409b5
Nov 02 12:19:37.584274 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:37.584250    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec1555fb-9336-4fb5-8dbf-11e4e95e23f4" path="/var/lib/kubelet/pods/ec1555fb-9336-4fb5-8dbf-11e4e95e23f4/volumes"
Nov 02 12:19:37.584715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:37.584698    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"]
Nov 02 12:19:38.055777 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:38.055748    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"]
Nov 02 12:19:38.086948 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:38.086914    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-460b9029-5c9j7"]
Nov 02 12:19:38.288301 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:38.288265    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr" event={"ID":"ad2bbef1-1336-4164-a083-c15cf501bfef","Type":"ContainerStarted","Data":"37e330362c10e1c1a650a3ed635a1f946ce769f8be2cc1fe2fc2e9c00a4804c7"}
Nov 02 12:19:38.288482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:38.288338    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr" event={"ID":"ad2bbef1-1336-4164-a083-c15cf501bfef","Type":"ContainerStarted","Data":"4b8e7e2aeeba5793b4e98c72d9d66ca0e42792dd7884642c4dd71325c50409b5"}
Nov 02 12:19:39.298420 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:39.298385    4464 generic.go:356] "Generic (PLEG): container finished" podID="ad2bbef1-1336-4164-a083-c15cf501bfef" containerID="37e330362c10e1c1a650a3ed635a1f946ce769f8be2cc1fe2fc2e9c00a4804c7" exitCode=0
Nov 02 12:19:39.298832 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:39.298462    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr" event={"ID":"ad2bbef1-1336-4164-a083-c15cf501bfef","Type":"ContainerDied","Data":"37e330362c10e1c1a650a3ed635a1f946ce769f8be2cc1fe2fc2e9c00a4804c7"}
Nov 02 12:19:39.583782 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:39.583710    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db3ca68-49f4-4ab1-b95d-4f34168c0a5c" path="/var/lib/kubelet/pods/6db3ca68-49f4-4ab1-b95d-4f34168c0a5c/volumes"
Nov 02 12:19:40.458451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:40.458425    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"
Nov 02 12:19:40.555043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:40.555013    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad2bbef1-1336-4164-a083-c15cf501bfef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2bbef1-1336-4164-a083-c15cf501bfef-maas-api-xks-ca\") pod \"ad2bbef1-1336-4164-a083-c15cf501bfef\" (UID: \"ad2bbef1-1336-4164-a083-c15cf501bfef\") "
Nov 02 12:19:40.555186 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:40.555133    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad2bbef1-1336-4164-a083-c15cf501bfef-kube-api-access-84lns\" (UniqueName: \"kubernetes.io/projected/ad2bbef1-1336-4164-a083-c15cf501bfef-kube-api-access-84lns\") pod \"ad2bbef1-1336-4164-a083-c15cf501bfef\" (UID: \"ad2bbef1-1336-4164-a083-c15cf501bfef\") "
Nov 02 12:19:40.555186 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:40.555180    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad2bbef1-1336-4164-a083-c15cf501bfef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2bbef1-1336-4164-a083-c15cf501bfef-maas-api-openshift-ca\") pod \"ad2bbef1-1336-4164-a083-c15cf501bfef\" (UID: \"ad2bbef1-1336-4164-a083-c15cf501bfef\") "
Nov 02 12:19:40.555595 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:40.555565    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2bbef1-1336-4164-a083-c15cf501bfef-maas-api-openshift-ca" pod "ad2bbef1-1336-4164-a083-c15cf501bfef" (UID: "ad2bbef1-1336-4164-a083-c15cf501bfef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:19:40.557091 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:40.557039    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad2bbef1-1336-4164-a083-c15cf501bfef-maas-api-xks-ca" pod "ad2bbef1-1336-4164-a083-c15cf501bfef" (UID: "ad2bbef1-1336-4164-a083-c15cf501bfef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:19:40.557180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:40.557118    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2bbef1-1336-4164-a083-c15cf501bfef-kube-api-access-84lns" pod "ad2bbef1-1336-4164-a083-c15cf501bfef" (UID: "ad2bbef1-1336-4164-a083-c15cf501bfef"). InnerVolumeSpecName "kube-api-access-84lns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:19:40.657138 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:40.657110    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2bbef1-1336-4164-a083-c15cf501bfef-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:40.657138 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:40.657136    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2bbef1-1336-4164-a083-c15cf501bfef-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:40.657138 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:40.657145    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84lns\" (UniqueName: \"kubernetes.io/projected/ad2bbef1-1336-4164-a083-c15cf501bfef-kube-api-access-84lns\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:41.316269 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:41.316240    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"
Nov 02 12:19:41.316445 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:41.316278    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr" event={"ID":"ad2bbef1-1336-4164-a083-c15cf501bfef","Type":"ContainerDied","Data":"4b8e7e2aeeba5793b4e98c72d9d66ca0e42792dd7884642c4dd71325c50409b5"}
Nov 02 12:19:41.316445 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:41.316324    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b8e7e2aeeba5793b4e98c72d9d66ca0e42792dd7884642c4dd71325c50409b5"
Nov 02 12:19:44.285581 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:44.285528    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:19:46.813105 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:46.813057    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v" podUID="01b8183e-899a-42ca-89b1-9a9db37b239a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:19:46.828005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:46.821062    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29" podUID="6ef3a987-e107-4351-9e85-cafeecf0a708" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:19:48.625182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:48.625117    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"]
Nov 02 12:19:48.626027 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:48.625970    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92" podUID="a1754568-1a2e-4c3a-9a37-2a80ef54ab00" containerName="istio-proxy" containerID="cri-o://bf42302a90351fedea4b1383ce3b220bdf49ac6643ab4181a04f90d51c495946" gracePeriod=30
Nov 02 12:19:52.828559 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:52.828527    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"
Nov 02 12:19:52.910787 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:52.910761    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"
Nov 02 12:19:52.981516 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:52.981483    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01b8183e-899a-42ca-89b1-9a9db37b239a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b8183e-899a-42ca-89b1-9a9db37b239a-plugins-config-volume\") pod \"01b8183e-899a-42ca-89b1-9a9db37b239a\" (UID: \"01b8183e-899a-42ca-89b1-9a9db37b239a\") "
Nov 02 12:19:52.981682 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:52.981532    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b8183e-899a-42ca-89b1-9a9db37b239a-kube-api-access-78cpd\" (UniqueName: \"kubernetes.io/projected/01b8183e-899a-42ca-89b1-9a9db37b239a-kube-api-access-78cpd\") pod \"01b8183e-899a-42ca-89b1-9a9db37b239a\" (UID: \"01b8183e-899a-42ca-89b1-9a9db37b239a\") "
Nov 02 12:19:52.981682 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:52.981661    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef3a987-e107-4351-9e85-cafeecf0a708-kube-api-access-pf9f2\" (UniqueName: \"kubernetes.io/projected/6ef3a987-e107-4351-9e85-cafeecf0a708-kube-api-access-pf9f2\") pod \"6ef3a987-e107-4351-9e85-cafeecf0a708\" (UID: \"6ef3a987-e107-4351-9e85-cafeecf0a708\") "
Nov 02 12:19:52.981762 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:52.981695    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef3a987-e107-4351-9e85-cafeecf0a708-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef3a987-e107-4351-9e85-cafeecf0a708-plugins-config-volume\") pod \"6ef3a987-e107-4351-9e85-cafeecf0a708\" (UID: \"6ef3a987-e107-4351-9e85-cafeecf0a708\") "
Nov 02 12:19:52.981959 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:52.981923    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b8183e-899a-42ca-89b1-9a9db37b239a-plugins-config-volume" pod "01b8183e-899a-42ca-89b1-9a9db37b239a" (UID: "01b8183e-899a-42ca-89b1-9a9db37b239a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:19:52.982157 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:52.982136    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef3a987-e107-4351-9e85-cafeecf0a708-plugins-config-volume" pod "6ef3a987-e107-4351-9e85-cafeecf0a708" (UID: "6ef3a987-e107-4351-9e85-cafeecf0a708"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:19:52.982290 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:52.982272    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef3a987-e107-4351-9e85-cafeecf0a708-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:52.982358 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:52.982296    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b8183e-899a-42ca-89b1-9a9db37b239a-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:52.984193 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:52.983914    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b8183e-899a-42ca-89b1-9a9db37b239a-kube-api-access-78cpd" pod "01b8183e-899a-42ca-89b1-9a9db37b239a" (UID: "01b8183e-899a-42ca-89b1-9a9db37b239a"). InnerVolumeSpecName "kube-api-access-78cpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:19:52.984193 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:52.984039    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef3a987-e107-4351-9e85-cafeecf0a708-kube-api-access-pf9f2" pod "6ef3a987-e107-4351-9e85-cafeecf0a708" (UID: "6ef3a987-e107-4351-9e85-cafeecf0a708"). InnerVolumeSpecName "kube-api-access-pf9f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:19:53.005458 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.005428    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"]
Nov 02 12:19:53.083592 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.083504    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78cpd\" (UniqueName: \"kubernetes.io/projected/01b8183e-899a-42ca-89b1-9a9db37b239a-kube-api-access-78cpd\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:53.083592 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.083535    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf9f2\" (UniqueName: \"kubernetes.io/projected/6ef3a987-e107-4351-9e85-cafeecf0a708-kube-api-access-pf9f2\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:53.578828 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.578799    4464 generic.go:356] "Generic (PLEG): container finished" podID="6ef3a987-e107-4351-9e85-cafeecf0a708" containerID="b430a2037a7be05b52705adcb988d5fbc414c4f65a6fe50aaf5f08b610f46dbf" exitCode=0
Nov 02 12:19:53.578963 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.578904    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"
Nov 02 12:19:53.580541 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.580514    4464 generic.go:356] "Generic (PLEG): container finished" podID="01b8183e-899a-42ca-89b1-9a9db37b239a" containerID="5265125648a116c355e29a8b6dd0975f34974a855a2acec498b8c2aaa8292a45" exitCode=0
Nov 02 12:19:53.580655 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.580600    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"
Nov 02 12:19:53.584222 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.584196    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29" event={"ID":"6ef3a987-e107-4351-9e85-cafeecf0a708","Type":"ContainerDied","Data":"b430a2037a7be05b52705adcb988d5fbc414c4f65a6fe50aaf5f08b610f46dbf"}
Nov 02 12:19:53.584341 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.584245    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29" event={"ID":"6ef3a987-e107-4351-9e85-cafeecf0a708","Type":"ContainerDied","Data":"1ac9cd7771bd1d45b16f5aabe327dceb63dc75a0334a10a673a8f895e8a136d2"}
Nov 02 12:19:53.584341 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.584270    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v" event={"ID":"01b8183e-899a-42ca-89b1-9a9db37b239a","Type":"ContainerDied","Data":"5265125648a116c355e29a8b6dd0975f34974a855a2acec498b8c2aaa8292a45"}
Nov 02 12:19:53.584341 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.584294    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v" event={"ID":"01b8183e-899a-42ca-89b1-9a9db37b239a","Type":"ContainerDied","Data":"2d49a310af7b8f72ed7f4e6810b8dd682342209c1633a82ba1d34f8ba309593f"}
Nov 02 12:19:53.584503 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.584353    4464 scope.go:122] "RemoveContainer" containerID="b430a2037a7be05b52705adcb988d5fbc414c4f65a6fe50aaf5f08b610f46dbf"
Nov 02 12:19:53.598906 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.598891    4464 scope.go:122] "RemoveContainer" containerID="b430a2037a7be05b52705adcb988d5fbc414c4f65a6fe50aaf5f08b610f46dbf"
Nov 02 12:19:53.599216 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:19:53.599195    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b430a2037a7be05b52705adcb988d5fbc414c4f65a6fe50aaf5f08b610f46dbf\": container with ID starting with b430a2037a7be05b52705adcb988d5fbc414c4f65a6fe50aaf5f08b610f46dbf not found: ID does not exist" containerID="b430a2037a7be05b52705adcb988d5fbc414c4f65a6fe50aaf5f08b610f46dbf"
Nov 02 12:19:53.599297 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.599223    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b430a2037a7be05b52705adcb988d5fbc414c4f65a6fe50aaf5f08b610f46dbf"} err="failed to get container status \"b430a2037a7be05b52705adcb988d5fbc414c4f65a6fe50aaf5f08b610f46dbf\": rpc error: code = NotFound desc = could not find container \"b430a2037a7be05b52705adcb988d5fbc414c4f65a6fe50aaf5f08b610f46dbf\": container with ID starting with b430a2037a7be05b52705adcb988d5fbc414c4f65a6fe50aaf5f08b610f46dbf not found: ID does not exist"
Nov 02 12:19:53.599297 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.599239    4464 scope.go:122] "RemoveContainer" containerID="5265125648a116c355e29a8b6dd0975f34974a855a2acec498b8c2aaa8292a45"
Nov 02 12:19:53.608750 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.608732    4464 scope.go:122] "RemoveContainer" containerID="5265125648a116c355e29a8b6dd0975f34974a855a2acec498b8c2aaa8292a45"
Nov 02 12:19:53.609019 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:19:53.609001    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5265125648a116c355e29a8b6dd0975f34974a855a2acec498b8c2aaa8292a45\": container with ID starting with 5265125648a116c355e29a8b6dd0975f34974a855a2acec498b8c2aaa8292a45 not found: ID does not exist" containerID="5265125648a116c355e29a8b6dd0975f34974a855a2acec498b8c2aaa8292a45"
Nov 02 12:19:53.609093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.609029    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5265125648a116c355e29a8b6dd0975f34974a855a2acec498b8c2aaa8292a45"} err="failed to get container status \"5265125648a116c355e29a8b6dd0975f34974a855a2acec498b8c2aaa8292a45\": rpc error: code = NotFound desc = could not find container \"5265125648a116c355e29a8b6dd0975f34974a855a2acec498b8c2aaa8292a45\": container with ID starting with 5265125648a116c355e29a8b6dd0975f34974a855a2acec498b8c2aaa8292a45 not found: ID does not exist"
Nov 02 12:19:53.757219 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.757176    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"]
Nov 02 12:19:53.762144 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.762118    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-036f88-6588f64457-jfg29"]
Nov 02 12:19:53.871833 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.871757    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"]
Nov 02 12:19:53.876116 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.876091    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-036f88-5b44fdb67-hhq4v"]
Nov 02 12:19:53.901203 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.901185    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:19:53.995088 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.995053    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-kube-api-access-nnjtd\" (UniqueName: \"kubernetes.io/projected/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-kube-api-access-nnjtd\") pod \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") "
Nov 02 12:19:53.995235 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.995097    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-token\" (UniqueName: \"kubernetes.io/projected/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-token\") pod \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") "
Nov 02 12:19:53.995235 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.995186    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-podinfo\") pod \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") "
Nov 02 12:19:53.995235 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.995228    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istiod-ca-cert\") pod \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") "
Nov 02 12:19:53.995353 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.995275    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-workload-certs\") pod \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") "
Nov 02 12:19:53.995385 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.995346    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-data\") pod \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") "
Nov 02 12:19:53.995550 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.995494    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-envoy\") pod \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") "
Nov 02 12:19:53.995726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.995563    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-workload-certs" pod "a1754568-1a2e-4c3a-9a37-2a80ef54ab00" (UID: "a1754568-1a2e-4c3a-9a37-2a80ef54ab00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:19:53.995726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.995605    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-credential-socket\") pod \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") "
Nov 02 12:19:53.995726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.995658    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-workload-socket\") pod \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\" (UID: \"a1754568-1a2e-4c3a-9a37-2a80ef54ab00\") "
Nov 02 12:19:53.995726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.995678    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-data" pod "a1754568-1a2e-4c3a-9a37-2a80ef54ab00" (UID: "a1754568-1a2e-4c3a-9a37-2a80ef54ab00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:19:53.995726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.995695    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istiod-ca-cert" pod "a1754568-1a2e-4c3a-9a37-2a80ef54ab00" (UID: "a1754568-1a2e-4c3a-9a37-2a80ef54ab00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:19:53.996032 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.995901    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-credential-socket" pod "a1754568-1a2e-4c3a-9a37-2a80ef54ab00" (UID: "a1754568-1a2e-4c3a-9a37-2a80ef54ab00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:19:53.996105 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.996065    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-workload-socket" pod "a1754568-1a2e-4c3a-9a37-2a80ef54ab00" (UID: "a1754568-1a2e-4c3a-9a37-2a80ef54ab00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:19:53.996264 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.996244    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:53.996331 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.996265    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:53.996331 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.996275    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:53.996331 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.996285    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:53.996331 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.996293    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:53.997490 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.997462    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-kube-api-access-nnjtd" pod "a1754568-1a2e-4c3a-9a37-2a80ef54ab00" (UID: "a1754568-1a2e-4c3a-9a37-2a80ef54ab00"). InnerVolumeSpecName "kube-api-access-nnjtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:19:53.997605 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.997531    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-envoy" pod "a1754568-1a2e-4c3a-9a37-2a80ef54ab00" (UID: "a1754568-1a2e-4c3a-9a37-2a80ef54ab00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:19:53.997712 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.997697    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-token" pod "a1754568-1a2e-4c3a-9a37-2a80ef54ab00" (UID: "a1754568-1a2e-4c3a-9a37-2a80ef54ab00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:19:53.997762 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:53.997699    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-podinfo" pod "a1754568-1a2e-4c3a-9a37-2a80ef54ab00" (UID: "a1754568-1a2e-4c3a-9a37-2a80ef54ab00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:19:54.097462 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:54.097423    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnjtd\" (UniqueName: \"kubernetes.io/projected/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-kube-api-access-nnjtd\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:54.097462 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:54.097456    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:54.097462 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:54.097469    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:54.097686 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:54.097481    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1754568-1a2e-4c3a-9a37-2a80ef54ab00-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:54.596890 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:54.596855    4464 generic.go:356] "Generic (PLEG): container finished" podID="a1754568-1a2e-4c3a-9a37-2a80ef54ab00" containerID="bf42302a90351fedea4b1383ce3b220bdf49ac6643ab4181a04f90d51c495946" exitCode=0
Nov 02 12:19:54.597083 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:54.596912    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"
Nov 02 12:19:54.597083 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:54.596928    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92" event={"ID":"a1754568-1a2e-4c3a-9a37-2a80ef54ab00","Type":"ContainerDied","Data":"bf42302a90351fedea4b1383ce3b220bdf49ac6643ab4181a04f90d51c495946"}
Nov 02 12:19:54.597083 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:54.596989    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92" event={"ID":"a1754568-1a2e-4c3a-9a37-2a80ef54ab00","Type":"ContainerDied","Data":"8aac853377bf62b43594265c9b7a902941747e07e765ada05ba858a586bb5a53"}
Nov 02 12:19:54.597083 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:54.597016    4464 scope.go:122] "RemoveContainer" containerID="bf42302a90351fedea4b1383ce3b220bdf49ac6643ab4181a04f90d51c495946"
Nov 02 12:19:54.607938 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:54.607919    4464 scope.go:122] "RemoveContainer" containerID="bf42302a90351fedea4b1383ce3b220bdf49ac6643ab4181a04f90d51c495946"
Nov 02 12:19:54.608215 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:19:54.608191    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf42302a90351fedea4b1383ce3b220bdf49ac6643ab4181a04f90d51c495946\": container with ID starting with bf42302a90351fedea4b1383ce3b220bdf49ac6643ab4181a04f90d51c495946 not found: ID does not exist" containerID="bf42302a90351fedea4b1383ce3b220bdf49ac6643ab4181a04f90d51c495946"
Nov 02 12:19:54.608276 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:54.608228    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf42302a90351fedea4b1383ce3b220bdf49ac6643ab4181a04f90d51c495946"} err="failed to get container status \"bf42302a90351fedea4b1383ce3b220bdf49ac6643ab4181a04f90d51c495946\": rpc error: code = NotFound desc = could not find container \"bf42302a90351fedea4b1383ce3b220bdf49ac6643ab4181a04f90d51c495946\": container with ID starting with bf42302a90351fedea4b1383ce3b220bdf49ac6643ab4181a04f90d51c495946 not found: ID does not exist"
Nov 02 12:19:54.802245 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:54.802205    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"]
Nov 02 12:19:54.824054 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:54.823475    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92"]
Nov 02 12:19:54.841568 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:54.841534    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mxp92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:19:54.841700 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:54.841610    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxp92" podUID="a1754568-1a2e-4c3a-9a37-2a80ef54ab00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:19:55.075737 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.075701    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"]
Nov 02 12:19:55.076395 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.076377    4464 cpu_manager.go:414] "containerMap: removing container" podUID="a1754568-1a2e-4c3a-9a37-2a80ef54ab00" containerName="istio-proxy"
Nov 02 12:19:55.076451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.076397    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1754568-1a2e-4c3a-9a37-2a80ef54ab00" containerName="istio-proxy"
Nov 02 12:19:55.076451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.076426    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6ef3a987-e107-4351-9e85-cafeecf0a708" containerName="payload-processing"
Nov 02 12:19:55.076451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.076431    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef3a987-e107-4351-9e85-cafeecf0a708" containerName="payload-processing"
Nov 02 12:19:55.076451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.076441    4464 cpu_manager.go:414] "containerMap: removing container" podUID="01b8183e-899a-42ca-89b1-9a9db37b239a" containerName="payload-pre-processing"
Nov 02 12:19:55.076451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.076447    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b8183e-899a-42ca-89b1-9a9db37b239a" containerName="payload-pre-processing"
Nov 02 12:19:55.076608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.076466    4464 cpu_manager.go:414] "containerMap: removing container" podUID="ad2bbef1-1336-4164-a083-c15cf501bfef" containerName="revoke-keys"
Nov 02 12:19:55.076608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.076471    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2bbef1-1336-4164-a083-c15cf501bfef" containerName="revoke-keys"
Nov 02 12:19:55.076608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.076559    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1754568-1a2e-4c3a-9a37-2a80ef54ab00" containerName="istio-proxy"
Nov 02 12:19:55.076608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.076573    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef3a987-e107-4351-9e85-cafeecf0a708" containerName="payload-processing"
Nov 02 12:19:55.076608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.076578    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b8183e-899a-42ca-89b1-9a9db37b239a" containerName="payload-pre-processing"
Nov 02 12:19:55.076608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.076587    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2bbef1-1336-4164-a083-c15cf501bfef" containerName="revoke-keys"
Nov 02 12:19:55.080480 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.080462    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"
Nov 02 12:19:55.097650 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.097627    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"]
Nov 02 12:19:55.101884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.101864    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"]
Nov 02 12:19:55.105457 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.105445    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f22fc2f8-pf9f5"]
Nov 02 12:19:55.108726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.108708    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-851771f0-jltsm"]
Nov 02 12:19:55.133806 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.133787    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6a432197-dockercfg-55vqj\""
Nov 02 12:19:55.133892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.133790    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6a432197\""
Nov 02 12:19:55.134448 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.134429    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"]
Nov 02 12:19:55.237002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.228786    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1252145-23ce-4c2c-bc8f-26fd85eb14f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g\" (UID: \"f1252145-23ce-4c2c-bc8f-26fd85eb14f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"
Nov 02 12:19:55.237002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.228868    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ms9\" (UniqueName: \"kubernetes.io/projected/f1252145-23ce-4c2c-bc8f-26fd85eb14f9-kube-api-access-v6ms9\") pod \"payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g\" (UID: \"f1252145-23ce-4c2c-bc8f-26fd85eb14f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"
Nov 02 12:19:55.268003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.267032    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"]
Nov 02 12:19:55.305349 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.305323    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"]
Nov 02 12:19:55.305552 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.305476    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"
Nov 02 12:19:55.329897 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.329834    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1252145-23ce-4c2c-bc8f-26fd85eb14f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g\" (UID: \"f1252145-23ce-4c2c-bc8f-26fd85eb14f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"
Nov 02 12:19:55.330033 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.329911    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ms9\" (UniqueName: \"kubernetes.io/projected/f1252145-23ce-4c2c-bc8f-26fd85eb14f9-kube-api-access-v6ms9\") pod \"payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g\" (UID: \"f1252145-23ce-4c2c-bc8f-26fd85eb14f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"
Nov 02 12:19:55.330661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.330637    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1252145-23ce-4c2c-bc8f-26fd85eb14f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g\" (UID: \"f1252145-23ce-4c2c-bc8f-26fd85eb14f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"
Nov 02 12:19:55.391997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.391945    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ms9\" (UniqueName: \"kubernetes.io/projected/f1252145-23ce-4c2c-bc8f-26fd85eb14f9-kube-api-access-v6ms9\") pod \"payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g\" (UID: \"f1252145-23ce-4c2c-bc8f-26fd85eb14f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"
Nov 02 12:19:55.393840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.393820    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"
Nov 02 12:19:55.431717 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.431686    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6fgr\" (UniqueName: \"kubernetes.io/projected/9c76562c-76e1-4367-a648-59f93a59e7ba-kube-api-access-f6fgr\") pod \"payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4\" (UID: \"9c76562c-76e1-4367-a648-59f93a59e7ba\") " pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"
Nov 02 12:19:55.431852 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.431802    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c76562c-76e1-4367-a648-59f93a59e7ba-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4\" (UID: \"9c76562c-76e1-4367-a648-59f93a59e7ba\") " pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"
Nov 02 12:19:55.532722 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.532690    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c76562c-76e1-4367-a648-59f93a59e7ba-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4\" (UID: \"9c76562c-76e1-4367-a648-59f93a59e7ba\") " pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"
Nov 02 12:19:55.532868 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.532823    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6fgr\" (UniqueName: \"kubernetes.io/projected/9c76562c-76e1-4367-a648-59f93a59e7ba-kube-api-access-f6fgr\") pod \"payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4\" (UID: \"9c76562c-76e1-4367-a648-59f93a59e7ba\") " pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"
Nov 02 12:19:55.533396 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.533375    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c76562c-76e1-4367-a648-59f93a59e7ba-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4\" (UID: \"9c76562c-76e1-4367-a648-59f93a59e7ba\") " pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"
Nov 02 12:19:55.582296 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.582232    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01b8183e-899a-42ca-89b1-9a9db37b239a" path="/var/lib/kubelet/pods/01b8183e-899a-42ca-89b1-9a9db37b239a/volumes"
Nov 02 12:19:55.582653 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.582639    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef3a987-e107-4351-9e85-cafeecf0a708" path="/var/lib/kubelet/pods/6ef3a987-e107-4351-9e85-cafeecf0a708/volumes"
Nov 02 12:19:55.583073 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.583060    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1754568-1a2e-4c3a-9a37-2a80ef54ab00" path="/var/lib/kubelet/pods/a1754568-1a2e-4c3a-9a37-2a80ef54ab00/volumes"
Nov 02 12:19:55.583736 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.583722    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9377375-ca57-48ab-936b-b918a9fc457c" path="/var/lib/kubelet/pods/a9377375-ca57-48ab-936b-b918a9fc457c/volumes"
Nov 02 12:19:55.584163 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.584147    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f21b7ce9-f251-4c0e-b820-0af71bfd13ce" path="/var/lib/kubelet/pods/f21b7ce9-f251-4c0e-b820-0af71bfd13ce/volumes"
Nov 02 12:19:55.595035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.595013    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6fgr\" (UniqueName: \"kubernetes.io/projected/9c76562c-76e1-4367-a648-59f93a59e7ba-kube-api-access-f6fgr\") pod \"payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4\" (UID: \"9c76562c-76e1-4367-a648-59f93a59e7ba\") " pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"
Nov 02 12:19:55.623589 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.623568    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"
Nov 02 12:19:55.778741 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:19:55.778709    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1252145_23ce_4c2c_bc8f_26fd85eb14f9.slice/crio-b5af3e7a1c87a47b3ec591af69090472b793a69176284d0cbbbbfe0ba2674e35 WatchSource:0}: Error finding container b5af3e7a1c87a47b3ec591af69090472b793a69176284d0cbbbbfe0ba2674e35: Status 404 returned error can't find the container with id b5af3e7a1c87a47b3ec591af69090472b793a69176284d0cbbbbfe0ba2674e35
Nov 02 12:19:55.780521 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:55.780489    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"]
Nov 02 12:19:56.002148 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:19:56.002118    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c76562c_76e1_4367_a648_59f93a59e7ba.slice/crio-a7730ecd9ed4b1b8137b92a0f0031fd15cba099c469129b729fa78910cbd08ea WatchSource:0}: Error finding container a7730ecd9ed4b1b8137b92a0f0031fd15cba099c469129b729fa78910cbd08ea: Status 404 returned error can't find the container with id a7730ecd9ed4b1b8137b92a0f0031fd15cba099c469129b729fa78910cbd08ea
Nov 02 12:19:56.002869 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:56.002848    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"]
Nov 02 12:19:56.618842 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:56.618802    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g" event={"ID":"f1252145-23ce-4c2c-bc8f-26fd85eb14f9","Type":"ContainerStarted","Data":"3b14d8086fa48f12f43942c6ae66f8a36e74d7d11fe0c7aedde9014ad32ade0d"}
Nov 02 12:19:56.619305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:56.618858    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g" event={"ID":"f1252145-23ce-4c2c-bc8f-26fd85eb14f9","Type":"ContainerStarted","Data":"b5af3e7a1c87a47b3ec591af69090472b793a69176284d0cbbbbfe0ba2674e35"}
Nov 02 12:19:56.619305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:56.618885    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"
Nov 02 12:19:56.620430 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:56.620398    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4" event={"ID":"9c76562c-76e1-4367-a648-59f93a59e7ba","Type":"ContainerStarted","Data":"96812a29a69c3e2bbf0e80ad7ce3056647a3cf88c3c73ec476ad25be501f5761"}
Nov 02 12:19:56.620564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:56.620451    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4" event={"ID":"9c76562c-76e1-4367-a648-59f93a59e7ba","Type":"ContainerStarted","Data":"a7730ecd9ed4b1b8137b92a0f0031fd15cba099c469129b729fa78910cbd08ea"}
Nov 02 12:19:56.620564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:56.620539    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"
Nov 02 12:19:56.739280 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:56.739234    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g" podStartSLOduration=1.739222855 podStartE2EDuration="1.739222855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:19:56.736130487 +0000 UTC m=+1425.299838961" watchObservedRunningTime="2026-11-02 12:19:56.739222855 +0000 UTC m=+1425.302931264"
Nov 02 12:19:56.859311 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:56.859233    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4" podStartSLOduration=1.859215083 podStartE2EDuration="1.859215083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:19:56.855532721 +0000 UTC m=+1425.419241131" watchObservedRunningTime="2026-11-02 12:19:56.859215083 +0000 UTC m=+1425.422923498"
Nov 02 12:19:58.006214 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.006175    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj" podUID="bcfe0979-1466-4266-aa52-ce7137f909cd" containerName="payload-processing" containerID="cri-o://25ab589686016372419c17aaee7fd9a3bd7a55451febd88c9cbff891dadb244e" gracePeriod=25
Nov 02 12:19:58.281352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.281290    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:58.297119 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:19:58.297088    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:19:58.368022 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.367990    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcfe0979-1466-4266-aa52-ce7137f909cd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcfe0979-1466-4266-aa52-ce7137f909cd-routing-overlay-volume\") pod \"bcfe0979-1466-4266-aa52-ce7137f909cd\" (UID: \"bcfe0979-1466-4266-aa52-ce7137f909cd\") "
Nov 02 12:19:58.368201 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.368094    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcfe0979-1466-4266-aa52-ce7137f909cd-kube-api-access-6zsql\" (UniqueName: \"kubernetes.io/projected/bcfe0979-1466-4266-aa52-ce7137f909cd-kube-api-access-6zsql\") pod \"bcfe0979-1466-4266-aa52-ce7137f909cd\" (UID: \"bcfe0979-1466-4266-aa52-ce7137f909cd\") "
Nov 02 12:19:58.368201 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.368139    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcfe0979-1466-4266-aa52-ce7137f909cd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcfe0979-1466-4266-aa52-ce7137f909cd-provider-credentials\") pod \"bcfe0979-1466-4266-aa52-ce7137f909cd\" (UID: \"bcfe0979-1466-4266-aa52-ce7137f909cd\") "
Nov 02 12:19:58.368348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.368227    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcfe0979-1466-4266-aa52-ce7137f909cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfe0979-1466-4266-aa52-ce7137f909cd-plugins-config-volume\") pod \"bcfe0979-1466-4266-aa52-ce7137f909cd\" (UID: \"bcfe0979-1466-4266-aa52-ce7137f909cd\") "
Nov 02 12:19:58.368348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.368325    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcfe0979-1466-4266-aa52-ce7137f909cd-routing-overlay-volume" pod "bcfe0979-1466-4266-aa52-ce7137f909cd" (UID: "bcfe0979-1466-4266-aa52-ce7137f909cd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:19:58.368630 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.368607    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcfe0979-1466-4266-aa52-ce7137f909cd-plugins-config-volume" pod "bcfe0979-1466-4266-aa52-ce7137f909cd" (UID: "bcfe0979-1466-4266-aa52-ce7137f909cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:19:58.368768 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.368692    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bcfe0979-1466-4266-aa52-ce7137f909cd-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:58.368768 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.368706    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfe0979-1466-4266-aa52-ce7137f909cd-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:58.370448 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.370427    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcfe0979-1466-4266-aa52-ce7137f909cd-provider-credentials" pod "bcfe0979-1466-4266-aa52-ce7137f909cd" (UID: "bcfe0979-1466-4266-aa52-ce7137f909cd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:19:58.370635 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.370612    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcfe0979-1466-4266-aa52-ce7137f909cd-kube-api-access-6zsql" pod "bcfe0979-1466-4266-aa52-ce7137f909cd" (UID: "bcfe0979-1466-4266-aa52-ce7137f909cd"). InnerVolumeSpecName "kube-api-access-6zsql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:19:58.470445 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.470412    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bcfe0979-1466-4266-aa52-ce7137f909cd-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:58.470445 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.470436    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zsql\" (UniqueName: \"kubernetes.io/projected/bcfe0979-1466-4266-aa52-ce7137f909cd-kube-api-access-6zsql\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:19:58.636454 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.636373    4464 generic.go:356] "Generic (PLEG): container finished" podID="bcfe0979-1466-4266-aa52-ce7137f909cd" containerID="25ab589686016372419c17aaee7fd9a3bd7a55451febd88c9cbff891dadb244e" exitCode=0
Nov 02 12:19:58.636454 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.636417    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"
Nov 02 12:19:58.636688 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.636458    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj" event={"ID":"bcfe0979-1466-4266-aa52-ce7137f909cd","Type":"ContainerDied","Data":"25ab589686016372419c17aaee7fd9a3bd7a55451febd88c9cbff891dadb244e"}
Nov 02 12:19:58.636688 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.636515    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj" event={"ID":"bcfe0979-1466-4266-aa52-ce7137f909cd","Type":"ContainerDied","Data":"929cd9561caf5df78c5a97447da5a160f5041e3e241330cba6abebc52cc955d5"}
Nov 02 12:19:58.636688 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.636547    4464 scope.go:122] "RemoveContainer" containerID="25ab589686016372419c17aaee7fd9a3bd7a55451febd88c9cbff891dadb244e"
Nov 02 12:19:58.648720 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.648704    4464 scope.go:122] "RemoveContainer" containerID="25ab589686016372419c17aaee7fd9a3bd7a55451febd88c9cbff891dadb244e"
Nov 02 12:19:58.648966 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:19:58.648946    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ab589686016372419c17aaee7fd9a3bd7a55451febd88c9cbff891dadb244e\": container with ID starting with 25ab589686016372419c17aaee7fd9a3bd7a55451febd88c9cbff891dadb244e not found: ID does not exist" containerID="25ab589686016372419c17aaee7fd9a3bd7a55451febd88c9cbff891dadb244e"
Nov 02 12:19:58.649038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.648999    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ab589686016372419c17aaee7fd9a3bd7a55451febd88c9cbff891dadb244e"} err="failed to get container status \"25ab589686016372419c17aaee7fd9a3bd7a55451febd88c9cbff891dadb244e\": rpc error: code = NotFound desc = could not find container \"25ab589686016372419c17aaee7fd9a3bd7a55451febd88c9cbff891dadb244e\": container with ID starting with 25ab589686016372419c17aaee7fd9a3bd7a55451febd88c9cbff891dadb244e not found: ID does not exist"
Nov 02 12:19:58.804924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.804895    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"]
Nov 02 12:19:58.855378 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:58.855339    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b96frhj"]
Nov 02 12:19:59.582313 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:19:59.582280    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcfe0979-1466-4266-aa52-ce7137f909cd" path="/var/lib/kubelet/pods/bcfe0979-1466-4266-aa52-ce7137f909cd/volumes"
Nov 02 12:20:03.476185 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:03.476108    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:20:03.476871 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:03.476839    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:20:06.134631 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.134585    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"]
Nov 02 12:20:06.135229 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.135213    4464 cpu_manager.go:414] "containerMap: removing container" podUID="bcfe0979-1466-4266-aa52-ce7137f909cd" containerName="payload-processing"
Nov 02 12:20:06.135293 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.135230    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcfe0979-1466-4266-aa52-ce7137f909cd" containerName="payload-processing"
Nov 02 12:20:06.135371 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.135360    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcfe0979-1466-4266-aa52-ce7137f909cd" containerName="payload-processing"
Nov 02 12:20:06.138952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.138938    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"
Nov 02 12:20:06.196054 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.196029    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"]
Nov 02 12:20:06.241150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.241124    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:20:06.241280 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.241188    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:20:06.241280 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.241262    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:20:06.241415 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.241403    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:20:06.248877 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.248853    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931ead32-c95e-43a9-b721-d3725cc713f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07b38599-whwmv\" (UID: \"931ead32-c95e-43a9-b721-d3725cc713f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"
Nov 02 12:20:06.248955 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.248932    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m786\" (UniqueName: \"kubernetes.io/projected/931ead32-c95e-43a9-b721-d3725cc713f4-kube-api-access-7m786\") pod \"maas-api-revoke-sub-07b38599-whwmv\" (UID: \"931ead32-c95e-43a9-b721-d3725cc713f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"
Nov 02 12:20:06.249082 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.249065    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931ead32-c95e-43a9-b721-d3725cc713f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07b38599-whwmv\" (UID: \"931ead32-c95e-43a9-b721-d3725cc713f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"
Nov 02 12:20:06.350671 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.350640    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931ead32-c95e-43a9-b721-d3725cc713f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07b38599-whwmv\" (UID: \"931ead32-c95e-43a9-b721-d3725cc713f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"
Nov 02 12:20:06.350822 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.350759    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m786\" (UniqueName: \"kubernetes.io/projected/931ead32-c95e-43a9-b721-d3725cc713f4-kube-api-access-7m786\") pod \"maas-api-revoke-sub-07b38599-whwmv\" (UID: \"931ead32-c95e-43a9-b721-d3725cc713f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"
Nov 02 12:20:06.350869 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.350838    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931ead32-c95e-43a9-b721-d3725cc713f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07b38599-whwmv\" (UID: \"931ead32-c95e-43a9-b721-d3725cc713f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"
Nov 02 12:20:06.351444 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.351414    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931ead32-c95e-43a9-b721-d3725cc713f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07b38599-whwmv\" (UID: \"931ead32-c95e-43a9-b721-d3725cc713f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"
Nov 02 12:20:06.353010 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.352969    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931ead32-c95e-43a9-b721-d3725cc713f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07b38599-whwmv\" (UID: \"931ead32-c95e-43a9-b721-d3725cc713f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"
Nov 02 12:20:06.412838 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.412778    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m786\" (UniqueName: \"kubernetes.io/projected/931ead32-c95e-43a9-b721-d3725cc713f4-kube-api-access-7m786\") pod \"maas-api-revoke-sub-07b38599-whwmv\" (UID: \"931ead32-c95e-43a9-b721-d3725cc713f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"
Nov 02 12:20:06.452618 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.452591    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"
Nov 02 12:20:06.831944 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:20:06.831914    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931ead32_c95e_43a9_b721_d3725cc713f4.slice/crio-b7050d9cd3040e08b16345124934b441250472b55c33243ba812c952d382c70c WatchSource:0}: Error finding container b7050d9cd3040e08b16345124934b441250472b55c33243ba812c952d382c70c: Status 404 returned error can't find the container with id b7050d9cd3040e08b16345124934b441250472b55c33243ba812c952d382c70c
Nov 02 12:20:06.832394 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.832372    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"]
Nov 02 12:20:06.863333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:06.863305    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv" event={"ID":"931ead32-c95e-43a9-b721-d3725cc713f4","Type":"ContainerStarted","Data":"b7050d9cd3040e08b16345124934b441250472b55c33243ba812c952d382c70c"}
Nov 02 12:20:07.631433 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:07.631401    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"
Nov 02 12:20:07.631896 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:07.631471    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"
Nov 02 12:20:07.875881 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:07.875842    4464 generic.go:356] "Generic (PLEG): container finished" podID="931ead32-c95e-43a9-b721-d3725cc713f4" containerID="a7c321d6f8d22b1ba5c53f54a05682639c28eba1251b9a932f96aa0100a07192" exitCode=0
Nov 02 12:20:07.876143 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:07.876121    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv" event={"ID":"931ead32-c95e-43a9-b721-d3725cc713f4","Type":"ContainerDied","Data":"a7c321d6f8d22b1ba5c53f54a05682639c28eba1251b9a932f96aa0100a07192"}
Nov 02 12:20:09.033489 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:09.033460    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"
Nov 02 12:20:09.092255 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:09.092232    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/931ead32-c95e-43a9-b721-d3725cc713f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931ead32-c95e-43a9-b721-d3725cc713f4-maas-api-openshift-ca\") pod \"931ead32-c95e-43a9-b721-d3725cc713f4\" (UID: \"931ead32-c95e-43a9-b721-d3725cc713f4\") "
Nov 02 12:20:09.092390 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:09.092272    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/931ead32-c95e-43a9-b721-d3725cc713f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931ead32-c95e-43a9-b721-d3725cc713f4-maas-api-xks-ca\") pod \"931ead32-c95e-43a9-b721-d3725cc713f4\" (UID: \"931ead32-c95e-43a9-b721-d3725cc713f4\") "
Nov 02 12:20:09.092390 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:09.092318    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/931ead32-c95e-43a9-b721-d3725cc713f4-kube-api-access-7m786\" (UniqueName: \"kubernetes.io/projected/931ead32-c95e-43a9-b721-d3725cc713f4-kube-api-access-7m786\") pod \"931ead32-c95e-43a9-b721-d3725cc713f4\" (UID: \"931ead32-c95e-43a9-b721-d3725cc713f4\") "
Nov 02 12:20:09.092592 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:09.092569    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931ead32-c95e-43a9-b721-d3725cc713f4-maas-api-openshift-ca" pod "931ead32-c95e-43a9-b721-d3725cc713f4" (UID: "931ead32-c95e-43a9-b721-d3725cc713f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:20:09.092814 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:09.092787    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/931ead32-c95e-43a9-b721-d3725cc713f4-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:09.094264 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:09.094241    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/931ead32-c95e-43a9-b721-d3725cc713f4-maas-api-xks-ca" pod "931ead32-c95e-43a9-b721-d3725cc713f4" (UID: "931ead32-c95e-43a9-b721-d3725cc713f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:20:09.094536 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:09.094517    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931ead32-c95e-43a9-b721-d3725cc713f4-kube-api-access-7m786" pod "931ead32-c95e-43a9-b721-d3725cc713f4" (UID: "931ead32-c95e-43a9-b721-d3725cc713f4"). InnerVolumeSpecName "kube-api-access-7m786". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:20:09.193579 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:09.193507    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/931ead32-c95e-43a9-b721-d3725cc713f4-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:09.193579 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:09.193539    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m786\" (UniqueName: \"kubernetes.io/projected/931ead32-c95e-43a9-b721-d3725cc713f4-kube-api-access-7m786\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:09.893342 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:09.893263    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"
Nov 02 12:20:09.893342 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:09.893293    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv" event={"ID":"931ead32-c95e-43a9-b721-d3725cc713f4","Type":"ContainerDied","Data":"b7050d9cd3040e08b16345124934b441250472b55c33243ba812c952d382c70c"}
Nov 02 12:20:09.893524 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:09.893344    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7050d9cd3040e08b16345124934b441250472b55c33243ba812c952d382c70c"
Nov 02 12:20:12.147296 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:12.147266    4464 scope.go:122] "RemoveContainer" containerID="039cd0614daace0a6981d555275c791de78e5403f95c50038a80e22be975c4ca"
Nov 02 12:20:12.157761 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:12.157740    4464 scope.go:122] "RemoveContainer" containerID="cd0f78235ba82900bfbd187072de3fb67d93bf61c9ab232dc6a78b48cc6748d3"
Nov 02 12:20:12.171425 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:12.171406    4464 scope.go:122] "RemoveContainer" containerID="46062b67fd056d75a348df435d04d7e396b5ab12f07b038afd59198d2625eba6"
Nov 02 12:20:12.180671 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:12.180656    4464 scope.go:122] "RemoveContainer" containerID="ab7cbdac041ad1d49affae1d4c1475e6660e41799930d2875299123c1ba41404"
Nov 02 12:20:17.610798 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:17.610764    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:20:19.394910 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:19.394873    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"]
Nov 02 12:20:19.395386 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:19.395112    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd" podUID="f9d165c8-5462-4659-9ca7-65df7b86f133" containerName="istio-proxy" containerID="cri-o://9a625e8af0f5676afcea662460c267046227d796a0a02c74be4c8794dc1d8363" gracePeriod=30
Nov 02 12:20:21.464362 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.464331    4464 patch_prober.go:29] interesting pod/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:20:21.464741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.464402    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd" podUID="f9d165c8-5462-4659-9ca7-65df7b86f133" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:20:21.615800 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.615765    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"]
Nov 02 12:20:21.616361 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.616347    4464 cpu_manager.go:414] "containerMap: removing container" podUID="931ead32-c95e-43a9-b721-d3725cc713f4" containerName="revoke-keys"
Nov 02 12:20:21.616420 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.616362    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="931ead32-c95e-43a9-b721-d3725cc713f4" containerName="revoke-keys"
Nov 02 12:20:21.616473 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.616463    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="931ead32-c95e-43a9-b721-d3725cc713f4" containerName="revoke-keys"
Nov 02 12:20:21.621797 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.621778    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"
Nov 02 12:20:21.659590 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.659562    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cc8e74b-fc5fj\" (UID: \"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"
Nov 02 12:20:21.659773 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.659750    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74484\" (UniqueName: \"kubernetes.io/projected/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-kube-api-access-74484\") pod \"maas-api-revoke-sub-6cc8e74b-fc5fj\" (UID: \"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"
Nov 02 12:20:21.659873 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.659857    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cc8e74b-fc5fj\" (UID: \"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"
Nov 02 12:20:21.675127 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.675108    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"]
Nov 02 12:20:21.720852 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.720828    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:20:21.720852 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.720828    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:20:21.721037 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.720833    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:20:21.721037 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.720838    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:20:21.760741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.760715    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cc8e74b-fc5fj\" (UID: \"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"
Nov 02 12:20:21.760841 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.760798    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74484\" (UniqueName: \"kubernetes.io/projected/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-kube-api-access-74484\") pod \"maas-api-revoke-sub-6cc8e74b-fc5fj\" (UID: \"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"
Nov 02 12:20:21.760882 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.760850    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cc8e74b-fc5fj\" (UID: \"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"
Nov 02 12:20:21.761469 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.761438    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cc8e74b-fc5fj\" (UID: \"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"
Nov 02 12:20:21.763114 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.763092    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cc8e74b-fc5fj\" (UID: \"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"
Nov 02 12:20:21.824138 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.824118    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74484\" (UniqueName: \"kubernetes.io/projected/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-kube-api-access-74484\") pod \"maas-api-revoke-sub-6cc8e74b-fc5fj\" (UID: \"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"
Nov 02 12:20:21.935176 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:21.935147    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"
Nov 02 12:20:22.342086 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:22.342007    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"]
Nov 02 12:20:22.347238 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:22.347223    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"
Nov 02 12:20:22.372248 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:22.368808    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33dbb82b-2601-4e00-883c-2b463feea237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-613d7352-pq8m7\" (UID: \"33dbb82b-2601-4e00-883c-2b463feea237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"
Nov 02 12:20:22.372248 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:22.368870    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33dbb82b-2601-4e00-883c-2b463feea237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-613d7352-pq8m7\" (UID: \"33dbb82b-2601-4e00-883c-2b463feea237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"
Nov 02 12:20:22.372248 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:22.368953    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bhxv\" (UniqueName: \"kubernetes.io/projected/33dbb82b-2601-4e00-883c-2b463feea237-kube-api-access-7bhxv\") pod \"maas-api-revoke-sub-613d7352-pq8m7\" (UID: \"33dbb82b-2601-4e00-883c-2b463feea237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"
Nov 02 12:20:22.372824 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:20:22.372796    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f69f9ba_c4ef_4444_b74f_b2d7f16c8852.slice/crio-3dd05db92ef6f549c27b4468e8b54d0d57a692a44e142928926cb084cc1adee9 WatchSource:0}: Error finding container 3dd05db92ef6f549c27b4468e8b54d0d57a692a44e142928926cb084cc1adee9: Status 404 returned error can't find the container with id 3dd05db92ef6f549c27b4468e8b54d0d57a692a44e142928926cb084cc1adee9
Nov 02 12:20:22.374570 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:22.374536    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"]
Nov 02 12:20:22.410023 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:22.409998    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"]
Nov 02 12:20:22.469836 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:22.469805    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bhxv\" (UniqueName: \"kubernetes.io/projected/33dbb82b-2601-4e00-883c-2b463feea237-kube-api-access-7bhxv\") pod \"maas-api-revoke-sub-613d7352-pq8m7\" (UID: \"33dbb82b-2601-4e00-883c-2b463feea237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"
Nov 02 12:20:22.470195 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:22.469925    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33dbb82b-2601-4e00-883c-2b463feea237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-613d7352-pq8m7\" (UID: \"33dbb82b-2601-4e00-883c-2b463feea237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"
Nov 02 12:20:22.470195 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:22.469952    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33dbb82b-2601-4e00-883c-2b463feea237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-613d7352-pq8m7\" (UID: \"33dbb82b-2601-4e00-883c-2b463feea237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"
Nov 02 12:20:22.470535 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:22.470514    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33dbb82b-2601-4e00-883c-2b463feea237-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-613d7352-pq8m7\" (UID: \"33dbb82b-2601-4e00-883c-2b463feea237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"
Nov 02 12:20:22.472163 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:22.472142    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33dbb82b-2601-4e00-883c-2b463feea237-maas-api-xks-ca\") pod \"maas-api-revoke-sub-613d7352-pq8m7\" (UID: \"33dbb82b-2601-4e00-883c-2b463feea237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"
Nov 02 12:20:22.531424 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:22.531398    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bhxv\" (UniqueName: \"kubernetes.io/projected/33dbb82b-2601-4e00-883c-2b463feea237-kube-api-access-7bhxv\") pod \"maas-api-revoke-sub-613d7352-pq8m7\" (UID: \"33dbb82b-2601-4e00-883c-2b463feea237\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"
Nov 02 12:20:22.661670 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:22.661611    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"
Nov 02 12:20:22.991523 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:22.991490    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj" event={"ID":"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852","Type":"ContainerStarted","Data":"3506d814c124158e8968f4ed6637214c093593adc92441a29a945f5c515d53ac"}
Nov 02 12:20:22.991719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:22.991546    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj" event={"ID":"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852","Type":"ContainerStarted","Data":"3dd05db92ef6f549c27b4468e8b54d0d57a692a44e142928926cb084cc1adee9"}
Nov 02 12:20:23.044743 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:23.044694    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"]
Nov 02 12:20:23.047494 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:20:23.047465    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33dbb82b_2601_4e00_883c_2b463feea237.slice/crio-cfe0c30126906e8d50ffb84eea0a75a26f44cda77c6f4e36722a827a20e0d83c WatchSource:0}: Error finding container cfe0c30126906e8d50ffb84eea0a75a26f44cda77c6f4e36722a827a20e0d83c: Status 404 returned error can't find the container with id cfe0c30126906e8d50ffb84eea0a75a26f44cda77c6f4e36722a827a20e0d83c
Nov 02 12:20:23.052546 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:23.052428    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"]
Nov 02 12:20:23.058735 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:23.058706    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e23f5b7-blz8x"]
Nov 02 12:20:23.115794 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:23.115748    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj" podStartSLOduration=2.115703802 podStartE2EDuration="2.115703802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:20:23.110961242 +0000 UTC m=+1451.674669652" watchObservedRunningTime="2026-11-02 12:20:23.115703802 +0000 UTC m=+1451.679412211"
Nov 02 12:20:23.582526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:23.582452    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202b9e62-26ac-4c0b-b479-e9469ba7e547" path="/var/lib/kubelet/pods/202b9e62-26ac-4c0b-b479-e9469ba7e547/volumes"
Nov 02 12:20:24.001034 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.000995    4464 generic.go:356] "Generic (PLEG): container finished" podID="33dbb82b-2601-4e00-883c-2b463feea237" containerID="7789532192a1afdbf2178293a8e7d4856567af4013b93e302fab696a67113228" exitCode=0
Nov 02 12:20:24.001213 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.001075    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7" event={"ID":"33dbb82b-2601-4e00-883c-2b463feea237","Type":"ContainerDied","Data":"7789532192a1afdbf2178293a8e7d4856567af4013b93e302fab696a67113228"}
Nov 02 12:20:24.001213 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.001137    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7" event={"ID":"33dbb82b-2601-4e00-883c-2b463feea237","Type":"ContainerStarted","Data":"cfe0c30126906e8d50ffb84eea0a75a26f44cda77c6f4e36722a827a20e0d83c"}
Nov 02 12:20:24.002692 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.002668    4464 generic.go:356] "Generic (PLEG): container finished" podID="1f69f9ba-c4ef-4444-b74f-b2d7f16c8852" containerID="3506d814c124158e8968f4ed6637214c093593adc92441a29a945f5c515d53ac" exitCode=0
Nov 02 12:20:24.002790 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.002745    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj" event={"ID":"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852","Type":"ContainerDied","Data":"3506d814c124158e8968f4ed6637214c093593adc92441a29a945f5c515d53ac"}
Nov 02 12:20:24.662813 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.662755    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:20:24.698225 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.698200    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d165c8-5462-4659-9ca7-65df7b86f133-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d165c8-5462-4659-9ca7-65df7b86f133-istiod-ca-cert\") pod \"f9d165c8-5462-4659-9ca7-65df7b86f133\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") "
Nov 02 12:20:24.698354 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.698314    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-credential-socket\") pod \"f9d165c8-5462-4659-9ca7-65df7b86f133\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") "
Nov 02 12:20:24.698354 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.698334    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-token\" (UniqueName: \"kubernetes.io/projected/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-token\") pod \"f9d165c8-5462-4659-9ca7-65df7b86f133\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") "
Nov 02 12:20:24.698513 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.698357    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-workload-certs\") pod \"f9d165c8-5462-4659-9ca7-65df7b86f133\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") "
Nov 02 12:20:24.698513 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.698373    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-podinfo\") pod \"f9d165c8-5462-4659-9ca7-65df7b86f133\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") "
Nov 02 12:20:24.698513 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.698406    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-envoy\") pod \"f9d165c8-5462-4659-9ca7-65df7b86f133\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") "
Nov 02 12:20:24.698513 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.698439    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d165c8-5462-4659-9ca7-65df7b86f133-kube-api-access-b2hj2\" (UniqueName: \"kubernetes.io/projected/f9d165c8-5462-4659-9ca7-65df7b86f133-kube-api-access-b2hj2\") pod \"f9d165c8-5462-4659-9ca7-65df7b86f133\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") "
Nov 02 12:20:24.698513 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.698507    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-workload-socket\") pod \"f9d165c8-5462-4659-9ca7-65df7b86f133\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") "
Nov 02 12:20:24.698747 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.698552    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-data\") pod \"f9d165c8-5462-4659-9ca7-65df7b86f133\" (UID: \"f9d165c8-5462-4659-9ca7-65df7b86f133\") "
Nov 02 12:20:24.698747 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.698603    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d165c8-5462-4659-9ca7-65df7b86f133-istiod-ca-cert" pod "f9d165c8-5462-4659-9ca7-65df7b86f133" (UID: "f9d165c8-5462-4659-9ca7-65df7b86f133"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:20:24.698747 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.698662    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-workload-certs" pod "f9d165c8-5462-4659-9ca7-65df7b86f133" (UID: "f9d165c8-5462-4659-9ca7-65df7b86f133"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:20:24.698747 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.698688    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-credential-socket" pod "f9d165c8-5462-4659-9ca7-65df7b86f133" (UID: "f9d165c8-5462-4659-9ca7-65df7b86f133"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:20:24.698924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.698856    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-workload-socket" pod "f9d165c8-5462-4659-9ca7-65df7b86f133" (UID: "f9d165c8-5462-4659-9ca7-65df7b86f133"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:20:24.698924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.698873    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-data" pod "f9d165c8-5462-4659-9ca7-65df7b86f133" (UID: "f9d165c8-5462-4659-9ca7-65df7b86f133"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:20:24.699082 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.699062    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:24.699210 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.699084    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:24.699210 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.699099    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:24.699210 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.699111    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:24.699210 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.699123    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d165c8-5462-4659-9ca7-65df7b86f133-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:24.700682 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.700659    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-envoy" pod "f9d165c8-5462-4659-9ca7-65df7b86f133" (UID: "f9d165c8-5462-4659-9ca7-65df7b86f133"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:20:24.700785 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.700746    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d165c8-5462-4659-9ca7-65df7b86f133-kube-api-access-b2hj2" pod "f9d165c8-5462-4659-9ca7-65df7b86f133" (UID: "f9d165c8-5462-4659-9ca7-65df7b86f133"). InnerVolumeSpecName "kube-api-access-b2hj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:20:24.701002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.700960    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-token" pod "f9d165c8-5462-4659-9ca7-65df7b86f133" (UID: "f9d165c8-5462-4659-9ca7-65df7b86f133"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:20:24.701002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.700967    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-podinfo" pod "f9d165c8-5462-4659-9ca7-65df7b86f133" (UID: "f9d165c8-5462-4659-9ca7-65df7b86f133"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:20:24.800822 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.800794    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:24.800822 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.800821    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:24.800822 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.800830    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d165c8-5462-4659-9ca7-65df7b86f133-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:24.801065 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:24.800839    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2hj2\" (UniqueName: \"kubernetes.io/projected/f9d165c8-5462-4659-9ca7-65df7b86f133-kube-api-access-b2hj2\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:25.011081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.011043    4464 generic.go:356] "Generic (PLEG): container finished" podID="f9d165c8-5462-4659-9ca7-65df7b86f133" containerID="9a625e8af0f5676afcea662460c267046227d796a0a02c74be4c8794dc1d8363" exitCode=0
Nov 02 12:20:25.011278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.011094    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"
Nov 02 12:20:25.011278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.011122    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd" event={"ID":"f9d165c8-5462-4659-9ca7-65df7b86f133","Type":"ContainerDied","Data":"9a625e8af0f5676afcea662460c267046227d796a0a02c74be4c8794dc1d8363"}
Nov 02 12:20:25.011278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.011176    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd" event={"ID":"f9d165c8-5462-4659-9ca7-65df7b86f133","Type":"ContainerDied","Data":"502480a2483a6b257a7c0a3f7927c72939d5f731af26b5648cbc5700ffe70a69"}
Nov 02 12:20:25.011278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.011198    4464 scope.go:122] "RemoveContainer" containerID="9a625e8af0f5676afcea662460c267046227d796a0a02c74be4c8794dc1d8363"
Nov 02 12:20:25.024098 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.024070    4464 scope.go:122] "RemoveContainer" containerID="9a625e8af0f5676afcea662460c267046227d796a0a02c74be4c8794dc1d8363"
Nov 02 12:20:25.024406 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:20:25.024383    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a625e8af0f5676afcea662460c267046227d796a0a02c74be4c8794dc1d8363\": container with ID starting with 9a625e8af0f5676afcea662460c267046227d796a0a02c74be4c8794dc1d8363 not found: ID does not exist" containerID="9a625e8af0f5676afcea662460c267046227d796a0a02c74be4c8794dc1d8363"
Nov 02 12:20:25.024509 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.024412    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a625e8af0f5676afcea662460c267046227d796a0a02c74be4c8794dc1d8363"} err="failed to get container status \"9a625e8af0f5676afcea662460c267046227d796a0a02c74be4c8794dc1d8363\": rpc error: code = NotFound desc = could not find container \"9a625e8af0f5676afcea662460c267046227d796a0a02c74be4c8794dc1d8363\": container with ID starting with 9a625e8af0f5676afcea662460c267046227d796a0a02c74be4c8794dc1d8363 not found: ID does not exist"
Nov 02 12:20:25.161496 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.161467    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"
Nov 02 12:20:25.181754 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.181727    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"]
Nov 02 12:20:25.187292 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.187267    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-prtjd"]
Nov 02 12:20:25.204966 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.204945    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"
Nov 02 12:20:25.205138 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.205121    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33dbb82b-2601-4e00-883c-2b463feea237-kube-api-access-7bhxv\" (UniqueName: \"kubernetes.io/projected/33dbb82b-2601-4e00-883c-2b463feea237-kube-api-access-7bhxv\") pod \"33dbb82b-2601-4e00-883c-2b463feea237\" (UID: \"33dbb82b-2601-4e00-883c-2b463feea237\") "
Nov 02 12:20:25.205284 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.205270    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33dbb82b-2601-4e00-883c-2b463feea237-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33dbb82b-2601-4e00-883c-2b463feea237-maas-api-xks-ca\") pod \"33dbb82b-2601-4e00-883c-2b463feea237\" (UID: \"33dbb82b-2601-4e00-883c-2b463feea237\") "
Nov 02 12:20:25.205322 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.205307    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33dbb82b-2601-4e00-883c-2b463feea237-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33dbb82b-2601-4e00-883c-2b463feea237-maas-api-openshift-ca\") pod \"33dbb82b-2601-4e00-883c-2b463feea237\" (UID: \"33dbb82b-2601-4e00-883c-2b463feea237\") "
Nov 02 12:20:25.205689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.205666    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33dbb82b-2601-4e00-883c-2b463feea237-maas-api-openshift-ca" pod "33dbb82b-2601-4e00-883c-2b463feea237" (UID: "33dbb82b-2601-4e00-883c-2b463feea237"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:20:25.207042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.207024    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33dbb82b-2601-4e00-883c-2b463feea237-kube-api-access-7bhxv" pod "33dbb82b-2601-4e00-883c-2b463feea237" (UID: "33dbb82b-2601-4e00-883c-2b463feea237"). InnerVolumeSpecName "kube-api-access-7bhxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:20:25.207245 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.207227    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33dbb82b-2601-4e00-883c-2b463feea237-maas-api-xks-ca" pod "33dbb82b-2601-4e00-883c-2b463feea237" (UID: "33dbb82b-2601-4e00-883c-2b463feea237"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:20:25.306383 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.306289    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-maas-api-openshift-ca\") pod \"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852\" (UID: \"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852\") "
Nov 02 12:20:25.306383 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.306361    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-maas-api-xks-ca\") pod \"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852\" (UID: \"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852\") "
Nov 02 12:20:25.306654 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.306537    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-kube-api-access-74484\" (UniqueName: \"kubernetes.io/projected/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-kube-api-access-74484\") pod \"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852\" (UID: \"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852\") "
Nov 02 12:20:25.307017 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.306990    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33dbb82b-2601-4e00-883c-2b463feea237-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:25.307381 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.307019    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33dbb82b-2601-4e00-883c-2b463feea237-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:25.307381 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.307038    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bhxv\" (UniqueName: \"kubernetes.io/projected/33dbb82b-2601-4e00-883c-2b463feea237-kube-api-access-7bhxv\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:25.307528 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.307501    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-maas-api-openshift-ca" pod "1f69f9ba-c4ef-4444-b74f-b2d7f16c8852" (UID: "1f69f9ba-c4ef-4444-b74f-b2d7f16c8852"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:20:25.309553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.309525    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-kube-api-access-74484" pod "1f69f9ba-c4ef-4444-b74f-b2d7f16c8852" (UID: "1f69f9ba-c4ef-4444-b74f-b2d7f16c8852"). InnerVolumeSpecName "kube-api-access-74484". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:20:25.311931 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.311903    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-maas-api-xks-ca" pod "1f69f9ba-c4ef-4444-b74f-b2d7f16c8852" (UID: "1f69f9ba-c4ef-4444-b74f-b2d7f16c8852"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:20:25.407860 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.407835    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74484\" (UniqueName: \"kubernetes.io/projected/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-kube-api-access-74484\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:25.407860 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.407860    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:25.407860 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.407871    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:25.592082 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:25.592004    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d165c8-5462-4659-9ca7-65df7b86f133" path="/var/lib/kubelet/pods/f9d165c8-5462-4659-9ca7-65df7b86f133/volumes"
Nov 02 12:20:26.019842 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:26.019811    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"
Nov 02 12:20:26.020239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:26.019839    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj" event={"ID":"1f69f9ba-c4ef-4444-b74f-b2d7f16c8852","Type":"ContainerDied","Data":"3dd05db92ef6f549c27b4468e8b54d0d57a692a44e142928926cb084cc1adee9"}
Nov 02 12:20:26.020239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:26.019894    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dd05db92ef6f549c27b4468e8b54d0d57a692a44e142928926cb084cc1adee9"
Nov 02 12:20:26.022669 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:26.022646    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7" event={"ID":"33dbb82b-2601-4e00-883c-2b463feea237","Type":"ContainerDied","Data":"cfe0c30126906e8d50ffb84eea0a75a26f44cda77c6f4e36722a827a20e0d83c"}
Nov 02 12:20:26.022789 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:26.022681    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfe0c30126906e8d50ffb84eea0a75a26f44cda77c6f4e36722a827a20e0d83c"
Nov 02 12:20:26.022789 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:26.022653    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"
Nov 02 12:20:28.048220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:28.048191    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"]
Nov 02 12:20:28.099057 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:28.099029    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf897492-bflt7"]
Nov 02 12:20:28.292711 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:20:28.292679    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:20:29.610999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:29.610272    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43fdfd1b-9092-4d87-8bff-e84eb05caaf0" path="/var/lib/kubelet/pods/43fdfd1b-9092-4d87-8bff-e84eb05caaf0/volumes"
Nov 02 12:20:29.913048 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:29.911864    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"]
Nov 02 12:20:29.924359 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:29.924315    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f9d165c8-5462-4659-9ca7-65df7b86f133" containerName="istio-proxy"
Nov 02 12:20:29.924569 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:29.924550    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d165c8-5462-4659-9ca7-65df7b86f133" containerName="istio-proxy"
Nov 02 12:20:29.924743 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:29.924726    4464 cpu_manager.go:414] "containerMap: removing container" podUID="33dbb82b-2601-4e00-883c-2b463feea237" containerName="revoke-keys"
Nov 02 12:20:29.924842 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:29.924828    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33dbb82b-2601-4e00-883c-2b463feea237" containerName="revoke-keys"
Nov 02 12:20:29.925306 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:29.925281    4464 cpu_manager.go:414] "containerMap: removing container" podUID="1f69f9ba-c4ef-4444-b74f-b2d7f16c8852" containerName="revoke-keys"
Nov 02 12:20:29.925565 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:29.925548    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f69f9ba-c4ef-4444-b74f-b2d7f16c8852" containerName="revoke-keys"
Nov 02 12:20:29.925839 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:29.925823    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="33dbb82b-2601-4e00-883c-2b463feea237" containerName="revoke-keys"
Nov 02 12:20:29.925950 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:29.925938    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f69f9ba-c4ef-4444-b74f-b2d7f16c8852" containerName="revoke-keys"
Nov 02 12:20:29.926060 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:29.926048    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d165c8-5462-4659-9ca7-65df7b86f133" containerName="istio-proxy"
Nov 02 12:20:29.949766 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:29.949735    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.249996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.246866    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n265m\" (UniqueName: \"kubernetes.io/projected/61efc388-83a8-4585-8399-815dea8f96bf-kube-api-access-n265m\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.249996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.246955    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-credential-socket\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.249996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.247013    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-istio-data\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.249996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.247048    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-istio-envoy\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.249996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.247081    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61efc388-83a8-4585-8399-815dea8f96bf-istiod-ca-cert\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.249996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.247126    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-workload-certs\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.249996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.247224    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-workload-socket\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.249996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.247414    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61efc388-83a8-4585-8399-815dea8f96bf-istio-token\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.249996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.247484    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61efc388-83a8-4585-8399-815dea8f96bf-istio-podinfo\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.261891 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.261867    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fbf1444a-openshift-default-dockercfg-7v49p\""
Nov 02 12:20:30.308002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.284553    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"]
Nov 02 12:20:30.392660 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.392618    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61efc388-83a8-4585-8399-815dea8f96bf-istio-podinfo\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.392915 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.392895    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n265m\" (UniqueName: \"kubernetes.io/projected/61efc388-83a8-4585-8399-815dea8f96bf-kube-api-access-n265m\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.445535 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.445498    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-credential-socket\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.445777 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.445752    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-istio-data\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.445900 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.445885    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-istio-envoy\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.446033 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.446016    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61efc388-83a8-4585-8399-815dea8f96bf-istiod-ca-cert\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.446181 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.446166    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-workload-certs\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.446372 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.446353    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-workload-socket\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.446653 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.446633    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61efc388-83a8-4585-8399-815dea8f96bf-istio-token\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.452424 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.452399    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-workload-certs\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.453922 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.453898    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-credential-socket\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.462091 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.454339    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-workload-socket\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.462091 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.454628    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-istio-data\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.462091 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.455295    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61efc388-83a8-4585-8399-815dea8f96bf-istiod-ca-cert\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.462091 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.458503    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-istio-envoy\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.482294 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.481695    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61efc388-83a8-4585-8399-815dea8f96bf-istio-podinfo\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.530275 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.530201    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n265m\" (UniqueName: \"kubernetes.io/projected/61efc388-83a8-4585-8399-815dea8f96bf-kube-api-access-n265m\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.545531 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.545500    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61efc388-83a8-4585-8399-815dea8f96bf-istio-token\") pod \"e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") " pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:30.772228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:30.772186    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:31.241415 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:20:31.241386    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61efc388_83a8_4585_8399_815dea8f96bf.slice/crio-1a67052a0a6c3060a63f91287e7575ef4a43fe6e3b33dac648ade538e82336dd WatchSource:0}: Error finding container 1a67052a0a6c3060a63f91287e7575ef4a43fe6e3b33dac648ade538e82336dd: Status 404 returned error can't find the container with id 1a67052a0a6c3060a63f91287e7575ef4a43fe6e3b33dac648ade538e82336dd
Nov 02 12:20:31.243403 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:31.243351    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"]
Nov 02 12:20:31.244240 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:31.244209    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:20:31.244329 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:31.244269    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:20:31.244329 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:31.244296    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:20:31.261671 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:31.261644    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt" event={"ID":"61efc388-83a8-4585-8399-815dea8f96bf","Type":"ContainerStarted","Data":"1a67052a0a6c3060a63f91287e7575ef4a43fe6e3b33dac648ade538e82336dd"}
Nov 02 12:20:32.270290 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:32.270252    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt" event={"ID":"61efc388-83a8-4585-8399-815dea8f96bf","Type":"ContainerStarted","Data":"ffeeca88a314e080740ea9fc8affcfa67df5a511afd3341e9ab03b38ef1662f8"}
Nov 02 12:20:32.443264 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:32.443210    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt" podStartSLOduration=3.443196504 podStartE2EDuration="3.443196504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:20:32.442687882 +0000 UTC m=+1461.006396311" watchObservedRunningTime="2026-11-02 12:20:32.443196504 +0000 UTC m=+1461.006904913"
Nov 02 12:20:32.773056 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:32.773025    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:32.778193 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:32.778167    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:33.054371 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.054294    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"]
Nov 02 12:20:33.058485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.058468    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"
Nov 02 12:20:33.112177 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.112038    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:20:33.112177 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.112055    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:20:33.112350 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.112243    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:20:33.112478 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.112459    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:20:33.116367 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.116343    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"]
Nov 02 12:20:33.224780 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.224744    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmktp\" (UniqueName: \"kubernetes.io/projected/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-kube-api-access-wmktp\") pod \"maas-api-revoke-sub-00690f22-7mfxc\" (UID: \"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"
Nov 02 12:20:33.224937 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.224784    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00690f22-7mfxc\" (UID: \"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"
Nov 02 12:20:33.225007 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.224942    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00690f22-7mfxc\" (UID: \"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"
Nov 02 12:20:33.277376 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.277349    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:33.278335 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.278315    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:20:33.326013 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.325927    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00690f22-7mfxc\" (UID: \"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"
Nov 02 12:20:33.326113 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.326053    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00690f22-7mfxc\" (UID: \"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"
Nov 02 12:20:33.326204 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.326192    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmktp\" (UniqueName: \"kubernetes.io/projected/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-kube-api-access-wmktp\") pod \"maas-api-revoke-sub-00690f22-7mfxc\" (UID: \"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"
Nov 02 12:20:33.326686 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.326662    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00690f22-7mfxc\" (UID: \"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"
Nov 02 12:20:33.328196 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.328177    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00690f22-7mfxc\" (UID: \"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"
Nov 02 12:20:33.390016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.389991    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmktp\" (UniqueName: \"kubernetes.io/projected/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-kube-api-access-wmktp\") pod \"maas-api-revoke-sub-00690f22-7mfxc\" (UID: \"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"
Nov 02 12:20:33.673153 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:33.673076    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"
Nov 02 12:20:34.055032 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:20:34.055000    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d1a43e_ac53_4ccf_9028_1d9dd232b0e7.slice/crio-64fdba3f2a4498d9d8f0b961027d4be95cd1b7b6a005b1be1f70910da0a46b43 WatchSource:0}: Error finding container 64fdba3f2a4498d9d8f0b961027d4be95cd1b7b6a005b1be1f70910da0a46b43: Status 404 returned error can't find the container with id 64fdba3f2a4498d9d8f0b961027d4be95cd1b7b6a005b1be1f70910da0a46b43
Nov 02 12:20:34.056504 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:34.056478    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"]
Nov 02 12:20:34.285531 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:34.285498    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc" event={"ID":"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7","Type":"ContainerStarted","Data":"64fdba3f2a4498d9d8f0b961027d4be95cd1b7b6a005b1be1f70910da0a46b43"}
Nov 02 12:20:35.294859 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:35.294827    4464 generic.go:356] "Generic (PLEG): container finished" podID="f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7" containerID="e01e3ad6dc9f3d4301cf53dc9bf1ad456e28369b045c4b3f9eb89f2b3d1d652a" exitCode=0
Nov 02 12:20:35.295276 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:35.294910    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc" event={"ID":"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7","Type":"ContainerDied","Data":"e01e3ad6dc9f3d4301cf53dc9bf1ad456e28369b045c4b3f9eb89f2b3d1d652a"}
Nov 02 12:20:36.455244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:36.455219    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"
Nov 02 12:20:36.567383 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:36.567312    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-maas-api-xks-ca\") pod \"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7\" (UID: \"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7\") "
Nov 02 12:20:36.567383 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:36.567363    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-maas-api-openshift-ca\") pod \"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7\" (UID: \"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7\") "
Nov 02 12:20:36.567560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:36.567411    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-kube-api-access-wmktp\" (UniqueName: \"kubernetes.io/projected/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-kube-api-access-wmktp\") pod \"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7\" (UID: \"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7\") "
Nov 02 12:20:36.567797 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:36.567772    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-maas-api-openshift-ca" pod "f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7" (UID: "f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:20:36.567930 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:36.567916    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:36.569485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:36.569456    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-maas-api-xks-ca" pod "f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7" (UID: "f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:20:36.569604 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:36.569572    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-kube-api-access-wmktp" pod "f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7" (UID: "f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7"). InnerVolumeSpecName "kube-api-access-wmktp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:20:36.669329 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:36.669300    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:36.669329 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:36.669327    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmktp\" (UniqueName: \"kubernetes.io/projected/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7-kube-api-access-wmktp\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:37.280552 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.280514    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"]
Nov 02 12:20:37.281405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.281382    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7" containerName="revoke-keys"
Nov 02 12:20:37.281405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.281403    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7" containerName="revoke-keys"
Nov 02 12:20:37.281788 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.281558    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7" containerName="revoke-keys"
Nov 02 12:20:37.285938 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.285917    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"
Nov 02 12:20:37.312534 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.312509    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"
Nov 02 12:20:37.312693 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.312509    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc" event={"ID":"f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7","Type":"ContainerDied","Data":"64fdba3f2a4498d9d8f0b961027d4be95cd1b7b6a005b1be1f70910da0a46b43"}
Nov 02 12:20:37.312693 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.312646    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64fdba3f2a4498d9d8f0b961027d4be95cd1b7b6a005b1be1f70910da0a46b43"
Nov 02 12:20:37.342456 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.342432    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"]
Nov 02 12:20:37.375900 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.375874    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e05943db-8502-44e9-a7de-4c3a635d2055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7fe11b7-lqtjh\" (UID: \"e05943db-8502-44e9-a7de-4c3a635d2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"
Nov 02 12:20:37.376042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.376027    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhdqq\" (UniqueName: \"kubernetes.io/projected/e05943db-8502-44e9-a7de-4c3a635d2055-kube-api-access-bhdqq\") pod \"maas-api-revoke-sub-c7fe11b7-lqtjh\" (UID: \"e05943db-8502-44e9-a7de-4c3a635d2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"
Nov 02 12:20:37.376093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.376078    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e05943db-8502-44e9-a7de-4c3a635d2055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7fe11b7-lqtjh\" (UID: \"e05943db-8502-44e9-a7de-4c3a635d2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"
Nov 02 12:20:37.476916 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.476889    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhdqq\" (UniqueName: \"kubernetes.io/projected/e05943db-8502-44e9-a7de-4c3a635d2055-kube-api-access-bhdqq\") pod \"maas-api-revoke-sub-c7fe11b7-lqtjh\" (UID: \"e05943db-8502-44e9-a7de-4c3a635d2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"
Nov 02 12:20:37.477268 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.476948    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e05943db-8502-44e9-a7de-4c3a635d2055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7fe11b7-lqtjh\" (UID: \"e05943db-8502-44e9-a7de-4c3a635d2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"
Nov 02 12:20:37.477268 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.477037    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e05943db-8502-44e9-a7de-4c3a635d2055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7fe11b7-lqtjh\" (UID: \"e05943db-8502-44e9-a7de-4c3a635d2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"
Nov 02 12:20:37.477624 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.477601    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e05943db-8502-44e9-a7de-4c3a635d2055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7fe11b7-lqtjh\" (UID: \"e05943db-8502-44e9-a7de-4c3a635d2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"
Nov 02 12:20:37.479356 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.479333    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e05943db-8502-44e9-a7de-4c3a635d2055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7fe11b7-lqtjh\" (UID: \"e05943db-8502-44e9-a7de-4c3a635d2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"
Nov 02 12:20:37.538454 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.538392    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhdqq\" (UniqueName: \"kubernetes.io/projected/e05943db-8502-44e9-a7de-4c3a635d2055-kube-api-access-bhdqq\") pod \"maas-api-revoke-sub-c7fe11b7-lqtjh\" (UID: \"e05943db-8502-44e9-a7de-4c3a635d2055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"
Nov 02 12:20:37.599829 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.599802    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"
Nov 02 12:20:37.999082 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:37.997613    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"]
Nov 02 12:20:37.999491 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:20:37.999445    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05943db_8502_44e9_a7de_4c3a635d2055.slice/crio-d0470f20d47c4161e69f678b16bd484a889a8fd7b511b2d79cfc2f7844cc217b WatchSource:0}: Error finding container d0470f20d47c4161e69f678b16bd484a889a8fd7b511b2d79cfc2f7844cc217b: Status 404 returned error can't find the container with id d0470f20d47c4161e69f678b16bd484a889a8fd7b511b2d79cfc2f7844cc217b
Nov 02 12:20:38.050888 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.050852    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"]
Nov 02 12:20:38.055234 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.055211    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57057b37-6rgvn"]
Nov 02 12:20:38.322227 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.322190    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh" event={"ID":"e05943db-8502-44e9-a7de-4c3a635d2055","Type":"ContainerStarted","Data":"a7dd4588aa990846e6ec2235e7bc034372f6eff1a72b59649cbfdc479d9426d0"}
Nov 02 12:20:38.322352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.322258    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh" event={"ID":"e05943db-8502-44e9-a7de-4c3a635d2055","Type":"ContainerStarted","Data":"d0470f20d47c4161e69f678b16bd484a889a8fd7b511b2d79cfc2f7844cc217b"}
Nov 02 12:20:38.394288 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.390367    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"]
Nov 02 12:20:38.444998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.443996    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"
Nov 02 12:20:38.452782 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.452750    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"]
Nov 02 12:20:38.472994 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.469373    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh" podStartSLOduration=1.469351682 podStartE2EDuration="1.469351682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:20:38.442475694 +0000 UTC m=+1467.006184103" watchObservedRunningTime="2026-11-02 12:20:38.469351682 +0000 UTC m=+1467.033060104"
Nov 02 12:20:38.493627 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.493591    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4khq\" (UniqueName: \"kubernetes.io/projected/8897bb5d-ac09-4389-a3fe-65bae125f6be-kube-api-access-l4khq\") pod \"payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2\" (UID: \"8897bb5d-ac09-4389-a3fe-65bae125f6be\") " pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"
Nov 02 12:20:38.493961 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.493909    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8897bb5d-ac09-4389-a3fe-65bae125f6be-plugins-config-volume\") pod \"payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2\" (UID: \"8897bb5d-ac09-4389-a3fe-65bae125f6be\") " pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"
Nov 02 12:20:38.497790 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.497757    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-fbf1444a-dockercfg-cw22q\""
Nov 02 12:20:38.497898 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.497875    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fbf1444a\""
Nov 02 12:20:38.525806 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.525746    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"]
Nov 02 12:20:38.530040 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.530018    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"
Nov 02 12:20:38.589092 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.589063    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"]
Nov 02 12:20:38.595214 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.595189    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2a8ed4-5986-40b4-a810-452e01712e38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt\" (UID: \"9c2a8ed4-5986-40b4-a810-452e01712e38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"
Nov 02 12:20:38.595342 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.595251    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57r2l\" (UniqueName: \"kubernetes.io/projected/9c2a8ed4-5986-40b4-a810-452e01712e38-kube-api-access-57r2l\") pod \"payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt\" (UID: \"9c2a8ed4-5986-40b4-a810-452e01712e38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"
Nov 02 12:20:38.595342 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.595312    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8897bb5d-ac09-4389-a3fe-65bae125f6be-plugins-config-volume\") pod \"payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2\" (UID: \"8897bb5d-ac09-4389-a3fe-65bae125f6be\") " pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"
Nov 02 12:20:38.596292 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.596267    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8897bb5d-ac09-4389-a3fe-65bae125f6be-plugins-config-volume\") pod \"payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2\" (UID: \"8897bb5d-ac09-4389-a3fe-65bae125f6be\") " pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"
Nov 02 12:20:38.596406 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.596326    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4khq\" (UniqueName: \"kubernetes.io/projected/8897bb5d-ac09-4389-a3fe-65bae125f6be-kube-api-access-l4khq\") pod \"payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2\" (UID: \"8897bb5d-ac09-4389-a3fe-65bae125f6be\") " pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"
Nov 02 12:20:38.658241 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.658213    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4khq\" (UniqueName: \"kubernetes.io/projected/8897bb5d-ac09-4389-a3fe-65bae125f6be-kube-api-access-l4khq\") pod \"payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2\" (UID: \"8897bb5d-ac09-4389-a3fe-65bae125f6be\") " pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"
Nov 02 12:20:38.697278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.697254    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2a8ed4-5986-40b4-a810-452e01712e38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt\" (UID: \"9c2a8ed4-5986-40b4-a810-452e01712e38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"
Nov 02 12:20:38.697387 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.697290    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57r2l\" (UniqueName: \"kubernetes.io/projected/9c2a8ed4-5986-40b4-a810-452e01712e38-kube-api-access-57r2l\") pod \"payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt\" (UID: \"9c2a8ed4-5986-40b4-a810-452e01712e38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"
Nov 02 12:20:38.697867 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.697848    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2a8ed4-5986-40b4-a810-452e01712e38-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt\" (UID: \"9c2a8ed4-5986-40b4-a810-452e01712e38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"
Nov 02 12:20:38.759074 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.759047    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57r2l\" (UniqueName: \"kubernetes.io/projected/9c2a8ed4-5986-40b4-a810-452e01712e38-kube-api-access-57r2l\") pod \"payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt\" (UID: \"9c2a8ed4-5986-40b4-a810-452e01712e38\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"
Nov 02 12:20:38.786148 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.786086    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"
Nov 02 12:20:38.845469 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:38.845440    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"
Nov 02 12:20:39.053895 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:39.053816    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"]
Nov 02 12:20:39.057909 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:39.057884    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ccc71d1-lr4jt"]
Nov 02 12:20:39.166238 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:39.166207    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"]
Nov 02 12:20:39.237735 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:20:39.237713    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2a8ed4_5986_40b4_a810_452e01712e38.slice/crio-c0ba11e2f3a788423caf74bee955faf9c2187cebafe3c5990a1497b3be1254e1 WatchSource:0}: Error finding container c0ba11e2f3a788423caf74bee955faf9c2187cebafe3c5990a1497b3be1254e1: Status 404 returned error can't find the container with id c0ba11e2f3a788423caf74bee955faf9c2187cebafe3c5990a1497b3be1254e1
Nov 02 12:20:39.241306 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:39.241282    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"]
Nov 02 12:20:39.340085 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:39.340010    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2" event={"ID":"8897bb5d-ac09-4389-a3fe-65bae125f6be","Type":"ContainerStarted","Data":"525de643b88494809697cc356e287e325a2b6f22ae51d517b1eaa694dc5183ec"}
Nov 02 12:20:39.340244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:39.340109    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2" event={"ID":"8897bb5d-ac09-4389-a3fe-65bae125f6be","Type":"ContainerStarted","Data":"041ee247d9e86b9c815a732c0ece01ca325f706db7008b10052eb3960761b29f"}
Nov 02 12:20:39.340244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:39.340201    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"
Nov 02 12:20:39.341964 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:39.341942    4464 generic.go:356] "Generic (PLEG): container finished" podID="e05943db-8502-44e9-a7de-4c3a635d2055" containerID="a7dd4588aa990846e6ec2235e7bc034372f6eff1a72b59649cbfdc479d9426d0" exitCode=0
Nov 02 12:20:39.342079 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:39.342017    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh" event={"ID":"e05943db-8502-44e9-a7de-4c3a635d2055","Type":"ContainerDied","Data":"a7dd4588aa990846e6ec2235e7bc034372f6eff1a72b59649cbfdc479d9426d0"}
Nov 02 12:20:39.343693 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:39.343674    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt" event={"ID":"9c2a8ed4-5986-40b4-a810-452e01712e38","Type":"ContainerStarted","Data":"1c6e9a3f63401fe79beb17ada2dce13e6fe1961baf65b0c55e7764b37b0055a5"}
Nov 02 12:20:39.343780 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:39.343712    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt" event={"ID":"9c2a8ed4-5986-40b4-a810-452e01712e38","Type":"ContainerStarted","Data":"c0ba11e2f3a788423caf74bee955faf9c2187cebafe3c5990a1497b3be1254e1"}
Nov 02 12:20:39.343780 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:39.343754    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"
Nov 02 12:20:39.460836 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:39.460786    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2" podStartSLOduration=1.460773018 podStartE2EDuration="1.460773018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:20:39.459296543 +0000 UTC m=+1468.023004953" watchObservedRunningTime="2026-11-02 12:20:39.460773018 +0000 UTC m=+1468.024481427"
Nov 02 12:20:39.584995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:39.583201    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt" podStartSLOduration=1.5831825579999999 podStartE2EDuration="1.583182558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:20:39.581899201 +0000 UTC m=+1468.145607612" watchObservedRunningTime="2026-11-02 12:20:39.583182558 +0000 UTC m=+1468.146890969"
Nov 02 12:20:39.613053 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:39.606797    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5fb801-5477-4f92-9ac7-8d65cf7df5d0" path="/var/lib/kubelet/pods/7d5fb801-5477-4f92-9ac7-8d65cf7df5d0/volumes"
Nov 02 12:20:39.613053 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:39.607516    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfbd370f-d4f3-440b-aa8d-e0d86efb9475" path="/var/lib/kubelet/pods/dfbd370f-d4f3-440b-aa8d-e0d86efb9475/volumes"
Nov 02 12:20:40.495051 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:40.495029    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"
Nov 02 12:20:40.621179 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:40.621102    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e05943db-8502-44e9-a7de-4c3a635d2055-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e05943db-8502-44e9-a7de-4c3a635d2055-maas-api-xks-ca\") pod \"e05943db-8502-44e9-a7de-4c3a635d2055\" (UID: \"e05943db-8502-44e9-a7de-4c3a635d2055\") "
Nov 02 12:20:40.622070 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:40.621210    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e05943db-8502-44e9-a7de-4c3a635d2055-kube-api-access-bhdqq\" (UniqueName: \"kubernetes.io/projected/e05943db-8502-44e9-a7de-4c3a635d2055-kube-api-access-bhdqq\") pod \"e05943db-8502-44e9-a7de-4c3a635d2055\" (UID: \"e05943db-8502-44e9-a7de-4c3a635d2055\") "
Nov 02 12:20:40.622070 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:40.621351    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e05943db-8502-44e9-a7de-4c3a635d2055-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e05943db-8502-44e9-a7de-4c3a635d2055-maas-api-openshift-ca\") pod \"e05943db-8502-44e9-a7de-4c3a635d2055\" (UID: \"e05943db-8502-44e9-a7de-4c3a635d2055\") "
Nov 02 12:20:40.622193 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:40.622113    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e05943db-8502-44e9-a7de-4c3a635d2055-maas-api-openshift-ca" pod "e05943db-8502-44e9-a7de-4c3a635d2055" (UID: "e05943db-8502-44e9-a7de-4c3a635d2055"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:20:40.623915 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:40.623874    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e05943db-8502-44e9-a7de-4c3a635d2055-maas-api-xks-ca" pod "e05943db-8502-44e9-a7de-4c3a635d2055" (UID: "e05943db-8502-44e9-a7de-4c3a635d2055"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:20:40.625316 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:40.625290    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e05943db-8502-44e9-a7de-4c3a635d2055-kube-api-access-bhdqq" pod "e05943db-8502-44e9-a7de-4c3a635d2055" (UID: "e05943db-8502-44e9-a7de-4c3a635d2055"). InnerVolumeSpecName "kube-api-access-bhdqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:20:40.723183 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:40.723153    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e05943db-8502-44e9-a7de-4c3a635d2055-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:40.723183 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:40.723183    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhdqq\" (UniqueName: \"kubernetes.io/projected/e05943db-8502-44e9-a7de-4c3a635d2055-kube-api-access-bhdqq\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:40.723373 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:40.723193    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e05943db-8502-44e9-a7de-4c3a635d2055-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:41.359424 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:41.359398    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"
Nov 02 12:20:41.359599 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:41.359425    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh" event={"ID":"e05943db-8502-44e9-a7de-4c3a635d2055","Type":"ContainerDied","Data":"d0470f20d47c4161e69f678b16bd484a889a8fd7b511b2d79cfc2f7844cc217b"}
Nov 02 12:20:41.359599 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:41.359471    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0470f20d47c4161e69f678b16bd484a889a8fd7b511b2d79cfc2f7844cc217b"
Nov 02 12:20:46.581630 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:46.581595    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:20:48.052824 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:48.052786    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"]
Nov 02 12:20:48.104108 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:48.104072    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9924385-kngsf"]
Nov 02 12:20:49.585038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:49.585008    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8363dde6-c7a7-422a-8d65-4999d48201d3" path="/var/lib/kubelet/pods/8363dde6-c7a7-422a-8d65-4999d48201d3/volumes"
Nov 02 12:20:50.353657 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:50.353622    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"
Nov 02 12:20:50.353840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:50.353698    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"
Nov 02 12:20:52.445580 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.445548    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"]
Nov 02 12:20:52.446156 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.446143    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e05943db-8502-44e9-a7de-4c3a635d2055" containerName="revoke-keys"
Nov 02 12:20:52.446212 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.446157    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e05943db-8502-44e9-a7de-4c3a635d2055" containerName="revoke-keys"
Nov 02 12:20:52.446297 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.446288    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e05943db-8502-44e9-a7de-4c3a635d2055" containerName="revoke-keys"
Nov 02 12:20:52.450723 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.450707    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"
Nov 02 12:20:52.482523 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.482503    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9lwx\" (UniqueName: \"kubernetes.io/projected/d8552f81-8b94-449f-9a45-db0690ae890e-kube-api-access-q9lwx\") pod \"maas-api-revoke-sub-a3635de3-72ccf\" (UID: \"d8552f81-8b94-449f-9a45-db0690ae890e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"
Nov 02 12:20:52.482644 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.482531    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8552f81-8b94-449f-9a45-db0690ae890e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3635de3-72ccf\" (UID: \"d8552f81-8b94-449f-9a45-db0690ae890e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"
Nov 02 12:20:52.482644 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.482556    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8552f81-8b94-449f-9a45-db0690ae890e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3635de3-72ccf\" (UID: \"d8552f81-8b94-449f-9a45-db0690ae890e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"
Nov 02 12:20:52.507950 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.507927    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"]
Nov 02 12:20:52.552297 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.552276    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:20:52.552523 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.552508    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:20:52.552564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.552532    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:20:52.554356 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.554342    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:20:52.583437 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.583415    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9lwx\" (UniqueName: \"kubernetes.io/projected/d8552f81-8b94-449f-9a45-db0690ae890e-kube-api-access-q9lwx\") pod \"maas-api-revoke-sub-a3635de3-72ccf\" (UID: \"d8552f81-8b94-449f-9a45-db0690ae890e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"
Nov 02 12:20:52.583586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.583442    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8552f81-8b94-449f-9a45-db0690ae890e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3635de3-72ccf\" (UID: \"d8552f81-8b94-449f-9a45-db0690ae890e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"
Nov 02 12:20:52.583586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.583462    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8552f81-8b94-449f-9a45-db0690ae890e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3635de3-72ccf\" (UID: \"d8552f81-8b94-449f-9a45-db0690ae890e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"
Nov 02 12:20:52.584158 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.584140    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8552f81-8b94-449f-9a45-db0690ae890e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3635de3-72ccf\" (UID: \"d8552f81-8b94-449f-9a45-db0690ae890e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"
Nov 02 12:20:52.585630 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.585610    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8552f81-8b94-449f-9a45-db0690ae890e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3635de3-72ccf\" (UID: \"d8552f81-8b94-449f-9a45-db0690ae890e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"
Nov 02 12:20:52.645022 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.644994    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9lwx\" (UniqueName: \"kubernetes.io/projected/d8552f81-8b94-449f-9a45-db0690ae890e-kube-api-access-q9lwx\") pod \"maas-api-revoke-sub-a3635de3-72ccf\" (UID: \"d8552f81-8b94-449f-9a45-db0690ae890e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"
Nov 02 12:20:52.764523 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:52.764500    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"
Nov 02 12:20:53.141423 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:20:53.141391    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8552f81_8b94_449f_9a45_db0690ae890e.slice/crio-dae213de93563893d7a8982e75603c5dab5b39933460c4c5602543fc0f5d6495 WatchSource:0}: Error finding container dae213de93563893d7a8982e75603c5dab5b39933460c4c5602543fc0f5d6495: Status 404 returned error can't find the container with id dae213de93563893d7a8982e75603c5dab5b39933460c4c5602543fc0f5d6495
Nov 02 12:20:53.142646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:53.142625    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"]
Nov 02 12:20:53.494712 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:53.494685    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf" event={"ID":"d8552f81-8b94-449f-9a45-db0690ae890e","Type":"ContainerStarted","Data":"4f4b58b98b105b7f18569c113ef8ab1567fb37b6c63394d24e0acdec8734c9c1"}
Nov 02 12:20:53.495051 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:53.494737    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf" event={"ID":"d8552f81-8b94-449f-9a45-db0690ae890e","Type":"ContainerStarted","Data":"dae213de93563893d7a8982e75603c5dab5b39933460c4c5602543fc0f5d6495"}
Nov 02 12:20:53.614307 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:53.614260    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf" podStartSLOduration=1.614245 podStartE2EDuration="1.614245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:20:53.611824577 +0000 UTC m=+1482.175532989" watchObservedRunningTime="2026-11-02 12:20:53.614245 +0000 UTC m=+1482.177953409"
Nov 02 12:20:54.503102 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:54.503069    4464 generic.go:356] "Generic (PLEG): container finished" podID="d8552f81-8b94-449f-9a45-db0690ae890e" containerID="4f4b58b98b105b7f18569c113ef8ab1567fb37b6c63394d24e0acdec8734c9c1" exitCode=0
Nov 02 12:20:54.503485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:54.503151    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf" event={"ID":"d8552f81-8b94-449f-9a45-db0690ae890e","Type":"ContainerDied","Data":"4f4b58b98b105b7f18569c113ef8ab1567fb37b6c63394d24e0acdec8734c9c1"}
Nov 02 12:20:55.050377 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:55.050345    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"]
Nov 02 12:20:55.054285 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:55.054263    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1649951e-zn8vp"]
Nov 02 12:20:55.584706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:55.584672    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f9bd26-5764-4ce1-99a3-f760b195a2f5" path="/var/lib/kubelet/pods/61f9bd26-5764-4ce1-99a3-f760b195a2f5/volumes"
Nov 02 12:20:55.664127 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:55.664104    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"
Nov 02 12:20:55.719526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:55.719504    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8552f81-8b94-449f-9a45-db0690ae890e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8552f81-8b94-449f-9a45-db0690ae890e-maas-api-xks-ca\") pod \"d8552f81-8b94-449f-9a45-db0690ae890e\" (UID: \"d8552f81-8b94-449f-9a45-db0690ae890e\") "
Nov 02 12:20:55.719672 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:55.719597    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8552f81-8b94-449f-9a45-db0690ae890e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8552f81-8b94-449f-9a45-db0690ae890e-maas-api-openshift-ca\") pod \"d8552f81-8b94-449f-9a45-db0690ae890e\" (UID: \"d8552f81-8b94-449f-9a45-db0690ae890e\") "
Nov 02 12:20:55.719800 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:55.719781    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8552f81-8b94-449f-9a45-db0690ae890e-kube-api-access-q9lwx\" (UniqueName: \"kubernetes.io/projected/d8552f81-8b94-449f-9a45-db0690ae890e-kube-api-access-q9lwx\") pod \"d8552f81-8b94-449f-9a45-db0690ae890e\" (UID: \"d8552f81-8b94-449f-9a45-db0690ae890e\") "
Nov 02 12:20:55.719971 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:55.719952    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8552f81-8b94-449f-9a45-db0690ae890e-maas-api-openshift-ca" pod "d8552f81-8b94-449f-9a45-db0690ae890e" (UID: "d8552f81-8b94-449f-9a45-db0690ae890e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:20:55.720264 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:55.720244    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8552f81-8b94-449f-9a45-db0690ae890e-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:55.721523 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:55.721504    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8552f81-8b94-449f-9a45-db0690ae890e-maas-api-xks-ca" pod "d8552f81-8b94-449f-9a45-db0690ae890e" (UID: "d8552f81-8b94-449f-9a45-db0690ae890e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:20:55.721722 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:55.721701    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8552f81-8b94-449f-9a45-db0690ae890e-kube-api-access-q9lwx" pod "d8552f81-8b94-449f-9a45-db0690ae890e" (UID: "d8552f81-8b94-449f-9a45-db0690ae890e"). InnerVolumeSpecName "kube-api-access-q9lwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:20:55.821078 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:55.821013    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8552f81-8b94-449f-9a45-db0690ae890e-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:55.821078 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:55.821041    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9lwx\" (UniqueName: \"kubernetes.io/projected/d8552f81-8b94-449f-9a45-db0690ae890e-kube-api-access-q9lwx\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:20:56.052044 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:56.052012    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"]
Nov 02 12:20:56.056011 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:56.055983    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a9451f-kl2hf"]
Nov 02 12:20:56.520516 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:56.520487    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"
Nov 02 12:20:56.520683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:56.520562    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf" event={"ID":"d8552f81-8b94-449f-9a45-db0690ae890e","Type":"ContainerDied","Data":"dae213de93563893d7a8982e75603c5dab5b39933460c4c5602543fc0f5d6495"}
Nov 02 12:20:56.520683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:56.520603    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dae213de93563893d7a8982e75603c5dab5b39933460c4c5602543fc0f5d6495"
Nov 02 12:20:57.582617 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:57.582577    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca53aea4-d372-4da2-b157-f74ef01dcfb7" path="/var/lib/kubelet/pods/ca53aea4-d372-4da2-b157-f74ef01dcfb7/volumes"
Nov 02 12:20:58.292839 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:20:58.292795    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:20:59.051745 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:59.051710    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"]
Nov 02 12:20:59.107215 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:59.107175    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-798c44bf-76g4d"]
Nov 02 12:20:59.617241 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:20:59.617206    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f495d2bb-beda-4dba-b98d-584df140c13b" path="/var/lib/kubelet/pods/f495d2bb-beda-4dba-b98d-584df140c13b/volumes"
Nov 02 12:21:05.048650 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:05.048617    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"]
Nov 02 12:21:05.098638 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:05.098606    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d10425d1-jql9m"]
Nov 02 12:21:05.583777 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:05.583743    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ae79f87-f000-4903-a930-aa9a5f12c324" path="/var/lib/kubelet/pods/8ae79f87-f000-4903-a930-aa9a5f12c324/volumes"
Nov 02 12:21:11.760264 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:11.760232    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:21:11.763844 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:11.763820    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:21:11.789042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:11.789013    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:21:11.791899 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:11.791881    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:21:12.049316 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:12.049229    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"]
Nov 02 12:21:12.100172 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:12.100142    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e3a7ce-7dt8z"]
Nov 02 12:21:12.251766 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:12.251734    4464 scope.go:122] "RemoveContainer" containerID="983ba93327f9001dc788e6901c7221ecbceee546733770cad30c6ebf122e1c78"
Nov 02 12:21:12.261907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:12.261888    4464 scope.go:122] "RemoveContainer" containerID="7104dadda3e3f3a43058179907dcdd6102412fb32286db2b7ab27f2bc64906f8"
Nov 02 12:21:12.270922 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:12.270900    4464 scope.go:122] "RemoveContainer" containerID="e6253d0e6275a71317cf57703e33262852f2f4618d67bc78ce0b64d52046a285"
Nov 02 12:21:12.279276 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:12.279258    4464 scope.go:122] "RemoveContainer" containerID="101421be2f46dbba9045b209493abd5ac2bc84e3e373843589a330d953f7d0e6"
Nov 02 12:21:12.291092 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:12.289092    4464 scope.go:122] "RemoveContainer" containerID="a0636e8fe4f2c58ffb52388ed50cc5dee6d41f84384ef7776828c0b6e40c316f"
Nov 02 12:21:13.582662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:13.582633    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53feb703-b468-42dc-aa83-8bca4e11a1c9" path="/var/lib/kubelet/pods/53feb703-b468-42dc-aa83-8bca4e11a1c9/volumes"
Nov 02 12:21:14.796741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:14.796703    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:21:17.096192 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.096158    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"]
Nov 02 12:21:17.096775 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.096762    4464 cpu_manager.go:414] "containerMap: removing container" podUID="d8552f81-8b94-449f-9a45-db0690ae890e" containerName="revoke-keys"
Nov 02 12:21:17.096816 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.096776    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8552f81-8b94-449f-9a45-db0690ae890e" containerName="revoke-keys"
Nov 02 12:21:17.096901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.096891    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8552f81-8b94-449f-9a45-db0690ae890e" containerName="revoke-keys"
Nov 02 12:21:17.102393 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.102375    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"
Nov 02 12:21:17.160781 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.160749    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"]
Nov 02 12:21:17.203503 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.203467    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:21:17.203656 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.203548    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:21:17.203656 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.203607    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:21:17.206145 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.206127    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:21:17.224994 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.224949    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cxrg\" (UniqueName: \"kubernetes.io/projected/fd1e6985-ca21-41cb-9bb4-442b25174e9f-kube-api-access-8cxrg\") pod \"maas-api-revoke-sub-0f88e997-t979s\" (UID: \"fd1e6985-ca21-41cb-9bb4-442b25174e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"
Nov 02 12:21:17.225141 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.225102    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1e6985-ca21-41cb-9bb4-442b25174e9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f88e997-t979s\" (UID: \"fd1e6985-ca21-41cb-9bb4-442b25174e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"
Nov 02 12:21:17.225250 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.225231    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1e6985-ca21-41cb-9bb4-442b25174e9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f88e997-t979s\" (UID: \"fd1e6985-ca21-41cb-9bb4-442b25174e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"
Nov 02 12:21:17.326929 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.326892    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1e6985-ca21-41cb-9bb4-442b25174e9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f88e997-t979s\" (UID: \"fd1e6985-ca21-41cb-9bb4-442b25174e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"
Nov 02 12:21:17.327123 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.327103    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cxrg\" (UniqueName: \"kubernetes.io/projected/fd1e6985-ca21-41cb-9bb4-442b25174e9f-kube-api-access-8cxrg\") pod \"maas-api-revoke-sub-0f88e997-t979s\" (UID: \"fd1e6985-ca21-41cb-9bb4-442b25174e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"
Nov 02 12:21:17.327204 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.327186    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1e6985-ca21-41cb-9bb4-442b25174e9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f88e997-t979s\" (UID: \"fd1e6985-ca21-41cb-9bb4-442b25174e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"
Nov 02 12:21:17.327853 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.327831    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1e6985-ca21-41cb-9bb4-442b25174e9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f88e997-t979s\" (UID: \"fd1e6985-ca21-41cb-9bb4-442b25174e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"
Nov 02 12:21:17.329352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.329334    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1e6985-ca21-41cb-9bb4-442b25174e9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f88e997-t979s\" (UID: \"fd1e6985-ca21-41cb-9bb4-442b25174e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"
Nov 02 12:21:17.388719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.388660    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cxrg\" (UniqueName: \"kubernetes.io/projected/fd1e6985-ca21-41cb-9bb4-442b25174e9f-kube-api-access-8cxrg\") pod \"maas-api-revoke-sub-0f88e997-t979s\" (UID: \"fd1e6985-ca21-41cb-9bb4-442b25174e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"
Nov 02 12:21:17.416503 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.416480    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"
Nov 02 12:21:17.794442 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:21:17.794413    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1e6985_ca21_41cb_9bb4_442b25174e9f.slice/crio-b33d150892f69155e4757b229b329117b536cfc59915daeb3a97e689eacc2a7f WatchSource:0}: Error finding container b33d150892f69155e4757b229b329117b536cfc59915daeb3a97e689eacc2a7f: Status 404 returned error can't find the container with id b33d150892f69155e4757b229b329117b536cfc59915daeb3a97e689eacc2a7f
Nov 02 12:21:17.795099 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.795075    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"]
Nov 02 12:21:17.854423 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:17.854391    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s" event={"ID":"fd1e6985-ca21-41cb-9bb4-442b25174e9f","Type":"ContainerStarted","Data":"b33d150892f69155e4757b229b329117b536cfc59915daeb3a97e689eacc2a7f"}
Nov 02 12:21:18.863386 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:18.863353    4464 generic.go:356] "Generic (PLEG): container finished" podID="fd1e6985-ca21-41cb-9bb4-442b25174e9f" containerID="79d417bc1334365c48a2cfc5b2551c7dd688bff1486e47fea6cd5d97dc21f4ad" exitCode=0
Nov 02 12:21:18.863770 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:18.863419    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s" event={"ID":"fd1e6985-ca21-41cb-9bb4-442b25174e9f","Type":"ContainerDied","Data":"79d417bc1334365c48a2cfc5b2551c7dd688bff1486e47fea6cd5d97dc21f4ad"}
Nov 02 12:21:20.073931 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:20.073886    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"
Nov 02 12:21:20.091340 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:20.091317    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd1e6985-ca21-41cb-9bb4-442b25174e9f-kube-api-access-8cxrg\" (UniqueName: \"kubernetes.io/projected/fd1e6985-ca21-41cb-9bb4-442b25174e9f-kube-api-access-8cxrg\") pod \"fd1e6985-ca21-41cb-9bb4-442b25174e9f\" (UID: \"fd1e6985-ca21-41cb-9bb4-442b25174e9f\") "
Nov 02 12:21:20.091506 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:20.091348    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd1e6985-ca21-41cb-9bb4-442b25174e9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1e6985-ca21-41cb-9bb4-442b25174e9f-maas-api-xks-ca\") pod \"fd1e6985-ca21-41cb-9bb4-442b25174e9f\" (UID: \"fd1e6985-ca21-41cb-9bb4-442b25174e9f\") "
Nov 02 12:21:20.091506 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:20.091387    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd1e6985-ca21-41cb-9bb4-442b25174e9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1e6985-ca21-41cb-9bb4-442b25174e9f-maas-api-openshift-ca\") pod \"fd1e6985-ca21-41cb-9bb4-442b25174e9f\" (UID: \"fd1e6985-ca21-41cb-9bb4-442b25174e9f\") "
Nov 02 12:21:20.091778 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:20.091755    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1e6985-ca21-41cb-9bb4-442b25174e9f-maas-api-openshift-ca" pod "fd1e6985-ca21-41cb-9bb4-442b25174e9f" (UID: "fd1e6985-ca21-41cb-9bb4-442b25174e9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:21:20.093461 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:20.093439    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1e6985-ca21-41cb-9bb4-442b25174e9f-maas-api-xks-ca" pod "fd1e6985-ca21-41cb-9bb4-442b25174e9f" (UID: "fd1e6985-ca21-41cb-9bb4-442b25174e9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:21:20.093532 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:20.093468    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1e6985-ca21-41cb-9bb4-442b25174e9f-kube-api-access-8cxrg" pod "fd1e6985-ca21-41cb-9bb4-442b25174e9f" (UID: "fd1e6985-ca21-41cb-9bb4-442b25174e9f"). InnerVolumeSpecName "kube-api-access-8cxrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:21:20.192611 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:20.192535    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cxrg\" (UniqueName: \"kubernetes.io/projected/fd1e6985-ca21-41cb-9bb4-442b25174e9f-kube-api-access-8cxrg\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:21:20.192611 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:20.192571    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1e6985-ca21-41cb-9bb4-442b25174e9f-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:21:20.192611 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:20.192586    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1e6985-ca21-41cb-9bb4-442b25174e9f-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:21:20.887578 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:20.887548    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"
Nov 02 12:21:20.887578 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:20.887544    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s" event={"ID":"fd1e6985-ca21-41cb-9bb4-442b25174e9f","Type":"ContainerDied","Data":"b33d150892f69155e4757b229b329117b536cfc59915daeb3a97e689eacc2a7f"}
Nov 02 12:21:20.887775 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:20.887647    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b33d150892f69155e4757b229b329117b536cfc59915daeb3a97e689eacc2a7f"
Nov 02 12:21:23.049806 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:23.049769    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"]
Nov 02 12:21:23.100224 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:23.100196    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-457b8470-sp5p8"]
Nov 02 12:21:23.582523 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:23.582485    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e14a726-d5db-43fa-9fa6-a697e7b6ec06" path="/var/lib/kubelet/pods/0e14a726-d5db-43fa-9fa6-a697e7b6ec06/volumes"
Nov 02 12:21:28.292115 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:21:28.292070    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:21:32.533058 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.533011    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"]
Nov 02 12:21:32.533666 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.533651    4464 cpu_manager.go:414] "containerMap: removing container" podUID="fd1e6985-ca21-41cb-9bb4-442b25174e9f" containerName="revoke-keys"
Nov 02 12:21:32.533718 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.533668    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd1e6985-ca21-41cb-9bb4-442b25174e9f" containerName="revoke-keys"
Nov 02 12:21:32.533777 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.533768    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd1e6985-ca21-41cb-9bb4-442b25174e9f" containerName="revoke-keys"
Nov 02 12:21:32.538300 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.538280    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"
Nov 02 12:21:32.595702 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.595672    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"]
Nov 02 12:21:32.609317 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.609286    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42bhq\" (UniqueName: \"kubernetes.io/projected/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-kube-api-access-42bhq\") pod \"maas-api-revoke-sub-7d40ee7f-kkthl\" (UID: \"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"
Nov 02 12:21:32.609449 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.609385    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d40ee7f-kkthl\" (UID: \"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"
Nov 02 12:21:32.609449 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.609423    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d40ee7f-kkthl\" (UID: \"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"
Nov 02 12:21:32.641208 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.641182    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:21:32.641328 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.641234    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:21:32.655211 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.655187    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:21:32.655211 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.655192    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:21:32.710689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.710658    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42bhq\" (UniqueName: \"kubernetes.io/projected/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-kube-api-access-42bhq\") pod \"maas-api-revoke-sub-7d40ee7f-kkthl\" (UID: \"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"
Nov 02 12:21:32.710840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.710729    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d40ee7f-kkthl\" (UID: \"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"
Nov 02 12:21:32.710840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.710752    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d40ee7f-kkthl\" (UID: \"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"
Nov 02 12:21:32.711481 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.711456    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d40ee7f-kkthl\" (UID: \"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"
Nov 02 12:21:32.712988 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.712947    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d40ee7f-kkthl\" (UID: \"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"
Nov 02 12:21:32.787104 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.787035    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42bhq\" (UniqueName: \"kubernetes.io/projected/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-kube-api-access-42bhq\") pod \"maas-api-revoke-sub-7d40ee7f-kkthl\" (UID: \"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"
Nov 02 12:21:32.852070 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:32.852037    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"
Nov 02 12:21:33.253585 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:21:33.253558    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cc04a4_dcc8_4eec_a73c_ed10d7867cb5.slice/crio-f9fa727af2832bf09f3d436a60ac5e5df11f18f6f10fac93ca6cbb8e52257f00 WatchSource:0}: Error finding container f9fa727af2832bf09f3d436a60ac5e5df11f18f6f10fac93ca6cbb8e52257f00: Status 404 returned error can't find the container with id f9fa727af2832bf09f3d436a60ac5e5df11f18f6f10fac93ca6cbb8e52257f00
Nov 02 12:21:33.253944 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:33.253922    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"]
Nov 02 12:21:34.017333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:34.017298    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl" event={"ID":"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5","Type":"ContainerStarted","Data":"facfca7dfa8f10ac50c87b4fd165b3df362b190df8be17f175c337594b2c42f9"}
Nov 02 12:21:34.017792 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:34.017363    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl" event={"ID":"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5","Type":"ContainerStarted","Data":"f9fa727af2832bf09f3d436a60ac5e5df11f18f6f10fac93ca6cbb8e52257f00"}
Nov 02 12:21:34.151621 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:34.151566    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl" podStartSLOduration=2.151553103 podStartE2EDuration="2.151553103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:21:34.148214295 +0000 UTC m=+1522.711922704" watchObservedRunningTime="2026-11-02 12:21:34.151553103 +0000 UTC m=+1522.715261513"
Nov 02 12:21:35.034793 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:35.034759    4464 generic.go:356] "Generic (PLEG): container finished" podID="b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5" containerID="facfca7dfa8f10ac50c87b4fd165b3df362b190df8be17f175c337594b2c42f9" exitCode=0
Nov 02 12:21:35.035194 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:35.034845    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl" event={"ID":"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5","Type":"ContainerDied","Data":"facfca7dfa8f10ac50c87b4fd165b3df362b190df8be17f175c337594b2c42f9"}
Nov 02 12:21:36.220637 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:36.220613    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"
Nov 02 12:21:36.353787 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:36.353690    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-maas-api-openshift-ca\") pod \"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5\" (UID: \"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5\") "
Nov 02 12:21:36.353787 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:36.353742    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-kube-api-access-42bhq\" (UniqueName: \"kubernetes.io/projected/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-kube-api-access-42bhq\") pod \"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5\" (UID: \"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5\") "
Nov 02 12:21:36.354093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:36.353806    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-maas-api-xks-ca\") pod \"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5\" (UID: \"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5\") "
Nov 02 12:21:36.354093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:36.354073    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-maas-api-openshift-ca" pod "b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5" (UID: "b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:21:36.354596 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:36.354413    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:21:36.357389 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:36.357354    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-maas-api-xks-ca" pod "b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5" (UID: "b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:21:36.357647 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:36.357627    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-kube-api-access-42bhq" pod "b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5" (UID: "b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5"). InnerVolumeSpecName "kube-api-access-42bhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:21:36.456317 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:36.456277    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:21:36.456317 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:36.456310    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42bhq\" (UniqueName: \"kubernetes.io/projected/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5-kube-api-access-42bhq\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:21:37.091007 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:37.090950    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"
Nov 02 12:21:37.091194 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:37.091003    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl" event={"ID":"b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5","Type":"ContainerDied","Data":"f9fa727af2832bf09f3d436a60ac5e5df11f18f6f10fac93ca6cbb8e52257f00"}
Nov 02 12:21:37.091194 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:37.091061    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9fa727af2832bf09f3d436a60ac5e5df11f18f6f10fac93ca6cbb8e52257f00"
Nov 02 12:21:39.107844 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:39.107806    4464 generic.go:356] "Generic (PLEG): container finished" podID="61efc388-83a8-4585-8399-815dea8f96bf" containerID="ffeeca88a314e080740ea9fc8affcfa67df5a511afd3341e9ab03b38ef1662f8" exitCode=0
Nov 02 12:21:39.108398 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:39.107876    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt" event={"ID":"61efc388-83a8-4585-8399-815dea8f96bf","Type":"ContainerDied","Data":"ffeeca88a314e080740ea9fc8affcfa67df5a511afd3341e9ab03b38ef1662f8"}
Nov 02 12:21:39.108398 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:39.108376    4464 scope.go:122] "RemoveContainer" containerID="ffeeca88a314e080740ea9fc8affcfa67df5a511afd3341e9ab03b38ef1662f8"
Nov 02 12:21:39.109782 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:39.109754    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:21:39.109892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:39.109834    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:21:39.109892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:39.109879    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:21:40.049847 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:40.049812    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"]
Nov 02 12:21:40.100408 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:40.100374    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63e8a4ce-fms2n"]
Nov 02 12:21:40.117493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:40.117461    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt" event={"ID":"61efc388-83a8-4585-8399-815dea8f96bf","Type":"ContainerStarted","Data":"6b89c66cecd382878d6cf2d193d6e4f8eb876399f7e471171827e73110b2a013"}
Nov 02 12:21:40.780359 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:40.780324    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:21:40.781769 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:40.781670    4464 patch_prober.go:29] interesting pod/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 12:21:40.781965 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:40.781943    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt" podUID="61efc388-83a8-4585-8399-815dea8f96bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 12:21:41.590174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:41.590138    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3e13b9b-3c31-43f2-abab-ddab6183e2fe" path="/var/lib/kubelet/pods/a3e13b9b-3c31-43f2-abab-ddab6183e2fe/volumes"
Nov 02 12:21:41.784000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:41.780326    4464 patch_prober.go:29] interesting pod/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 12:21:41.784000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:41.780387    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt" podUID="61efc388-83a8-4585-8399-815dea8f96bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 12:21:42.779117 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:42.779079    4464 patch_prober.go:29] interesting pod/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 12:21:42.779691 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:42.779659    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt" podUID="61efc388-83a8-4585-8399-815dea8f96bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 12:21:43.779007 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:43.778326    4464 patch_prober.go:29] interesting pod/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 12:21:43.779007 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:43.778387    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt" podUID="61efc388-83a8-4585-8399-815dea8f96bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 12:21:44.782818 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:44.782783    4464 patch_prober.go:29] interesting pod/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 12:21:44.783406 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:44.783376    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt" podUID="61efc388-83a8-4585-8399-815dea8f96bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 12:21:45.777768 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:45.777736    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:21:45.778218 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:45.778195    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:21:45.778796 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:45.778779    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:21:46.538644 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.538613    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"]
Nov 02 12:21:46.539297 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.539281    4464 cpu_manager.go:414] "containerMap: removing container" podUID="b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5" containerName="revoke-keys"
Nov 02 12:21:46.539355 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.539298    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5" containerName="revoke-keys"
Nov 02 12:21:46.539403 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.539394    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5" containerName="revoke-keys"
Nov 02 12:21:46.544327 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.544303    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"
Nov 02 12:21:46.597773 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.597745    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:21:46.597908 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.597789    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:21:46.597908 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.597827    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:21:46.598248 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.598226    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:21:46.601406 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.601386    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"]
Nov 02 12:21:46.620379 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.620355    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncprs\" (UniqueName: \"kubernetes.io/projected/aedbeb68-0cf8-48c4-a5e8-780c5db76876-kube-api-access-ncprs\") pod \"maas-api-revoke-sub-34ef7c53-ktlpg\" (UID: \"aedbeb68-0cf8-48c4-a5e8-780c5db76876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"
Nov 02 12:21:46.620590 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.620555    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aedbeb68-0cf8-48c4-a5e8-780c5db76876-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ef7c53-ktlpg\" (UID: \"aedbeb68-0cf8-48c4-a5e8-780c5db76876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"
Nov 02 12:21:46.620767 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.620738    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aedbeb68-0cf8-48c4-a5e8-780c5db76876-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ef7c53-ktlpg\" (UID: \"aedbeb68-0cf8-48c4-a5e8-780c5db76876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"
Nov 02 12:21:46.721766 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.721736    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aedbeb68-0cf8-48c4-a5e8-780c5db76876-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ef7c53-ktlpg\" (UID: \"aedbeb68-0cf8-48c4-a5e8-780c5db76876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"
Nov 02 12:21:46.721932 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.721834    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aedbeb68-0cf8-48c4-a5e8-780c5db76876-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ef7c53-ktlpg\" (UID: \"aedbeb68-0cf8-48c4-a5e8-780c5db76876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"
Nov 02 12:21:46.721932 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.721909    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncprs\" (UniqueName: \"kubernetes.io/projected/aedbeb68-0cf8-48c4-a5e8-780c5db76876-kube-api-access-ncprs\") pod \"maas-api-revoke-sub-34ef7c53-ktlpg\" (UID: \"aedbeb68-0cf8-48c4-a5e8-780c5db76876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"
Nov 02 12:21:46.722426 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.722405    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aedbeb68-0cf8-48c4-a5e8-780c5db76876-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ef7c53-ktlpg\" (UID: \"aedbeb68-0cf8-48c4-a5e8-780c5db76876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"
Nov 02 12:21:46.724283 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.724257    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aedbeb68-0cf8-48c4-a5e8-780c5db76876-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ef7c53-ktlpg\" (UID: \"aedbeb68-0cf8-48c4-a5e8-780c5db76876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"
Nov 02 12:21:46.783425 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.783400    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncprs\" (UniqueName: \"kubernetes.io/projected/aedbeb68-0cf8-48c4-a5e8-780c5db76876-kube-api-access-ncprs\") pod \"maas-api-revoke-sub-34ef7c53-ktlpg\" (UID: \"aedbeb68-0cf8-48c4-a5e8-780c5db76876\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"
Nov 02 12:21:46.858173 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:46.858101    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"
Nov 02 12:21:47.243810 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:21:47.243780    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaedbeb68_0cf8_48c4_a5e8_780c5db76876.slice/crio-0090b8da34fcecf1e2ebf0ba2ff2776ba79a81b3f5ae9e4d69e30ecc3deca5b3 WatchSource:0}: Error finding container 0090b8da34fcecf1e2ebf0ba2ff2776ba79a81b3f5ae9e4d69e30ecc3deca5b3: Status 404 returned error can't find the container with id 0090b8da34fcecf1e2ebf0ba2ff2776ba79a81b3f5ae9e4d69e30ecc3deca5b3
Nov 02 12:21:47.244150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:47.244133    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"]
Nov 02 12:21:48.186366 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:48.186334    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg" event={"ID":"aedbeb68-0cf8-48c4-a5e8-780c5db76876","Type":"ContainerStarted","Data":"8f5691584389afe8c5066da584265c0521528cdffbdbc4757aacf9af759f2c36"}
Nov 02 12:21:48.186778 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:48.186388    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg" event={"ID":"aedbeb68-0cf8-48c4-a5e8-780c5db76876","Type":"ContainerStarted","Data":"0090b8da34fcecf1e2ebf0ba2ff2776ba79a81b3f5ae9e4d69e30ecc3deca5b3"}
Nov 02 12:21:48.319295 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:48.319249    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg" podStartSLOduration=2.319235069 podStartE2EDuration="2.319235069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:21:48.315396654 +0000 UTC m=+1536.879105064" watchObservedRunningTime="2026-11-02 12:21:48.319235069 +0000 UTC m=+1536.882943499"
Nov 02 12:21:48.937795 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:48.937752    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:21:49.195738 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:49.195652    4464 generic.go:356] "Generic (PLEG): container finished" podID="aedbeb68-0cf8-48c4-a5e8-780c5db76876" containerID="8f5691584389afe8c5066da584265c0521528cdffbdbc4757aacf9af759f2c36" exitCode=0
Nov 02 12:21:49.196110 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:49.195731    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg" event={"ID":"aedbeb68-0cf8-48c4-a5e8-780c5db76876","Type":"ContainerDied","Data":"8f5691584389afe8c5066da584265c0521528cdffbdbc4757aacf9af759f2c36"}
Nov 02 12:21:50.348168 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:50.348144    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"
Nov 02 12:21:50.372563 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:50.372537    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aedbeb68-0cf8-48c4-a5e8-780c5db76876-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aedbeb68-0cf8-48c4-a5e8-780c5db76876-maas-api-openshift-ca\") pod \"aedbeb68-0cf8-48c4-a5e8-780c5db76876\" (UID: \"aedbeb68-0cf8-48c4-a5e8-780c5db76876\") "
Nov 02 12:21:50.372691 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:50.372579    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aedbeb68-0cf8-48c4-a5e8-780c5db76876-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aedbeb68-0cf8-48c4-a5e8-780c5db76876-maas-api-xks-ca\") pod \"aedbeb68-0cf8-48c4-a5e8-780c5db76876\" (UID: \"aedbeb68-0cf8-48c4-a5e8-780c5db76876\") "
Nov 02 12:21:50.372729 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:50.372705    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aedbeb68-0cf8-48c4-a5e8-780c5db76876-kube-api-access-ncprs\" (UniqueName: \"kubernetes.io/projected/aedbeb68-0cf8-48c4-a5e8-780c5db76876-kube-api-access-ncprs\") pod \"aedbeb68-0cf8-48c4-a5e8-780c5db76876\" (UID: \"aedbeb68-0cf8-48c4-a5e8-780c5db76876\") "
Nov 02 12:21:50.372928 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:50.372907    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aedbeb68-0cf8-48c4-a5e8-780c5db76876-maas-api-openshift-ca" pod "aedbeb68-0cf8-48c4-a5e8-780c5db76876" (UID: "aedbeb68-0cf8-48c4-a5e8-780c5db76876"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:21:50.373153 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:50.373133    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aedbeb68-0cf8-48c4-a5e8-780c5db76876-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:21:50.374587 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:50.374556    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aedbeb68-0cf8-48c4-a5e8-780c5db76876-maas-api-xks-ca" pod "aedbeb68-0cf8-48c4-a5e8-780c5db76876" (UID: "aedbeb68-0cf8-48c4-a5e8-780c5db76876"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:21:50.374671 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:50.374633    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aedbeb68-0cf8-48c4-a5e8-780c5db76876-kube-api-access-ncprs" pod "aedbeb68-0cf8-48c4-a5e8-780c5db76876" (UID: "aedbeb68-0cf8-48c4-a5e8-780c5db76876"). InnerVolumeSpecName "kube-api-access-ncprs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:21:50.474050 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:50.474022    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncprs\" (UniqueName: \"kubernetes.io/projected/aedbeb68-0cf8-48c4-a5e8-780c5db76876-kube-api-access-ncprs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:21:50.474178 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:50.474056    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aedbeb68-0cf8-48c4-a5e8-780c5db76876-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:21:51.211462 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:51.211431    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"
Nov 02 12:21:51.211639 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:51.211482    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg" event={"ID":"aedbeb68-0cf8-48c4-a5e8-780c5db76876","Type":"ContainerDied","Data":"0090b8da34fcecf1e2ebf0ba2ff2776ba79a81b3f5ae9e4d69e30ecc3deca5b3"}
Nov 02 12:21:51.211639 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:51.211540    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0090b8da34fcecf1e2ebf0ba2ff2776ba79a81b3f5ae9e4d69e30ecc3deca5b3"
Nov 02 12:21:54.049359 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:54.049324    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"]
Nov 02 12:21:54.099566 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:54.099536    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-370e6bbb-zk8wh"]
Nov 02 12:21:55.582020 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:55.581988    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd1a779-5e7b-45df-890c-b759b3b17f3c" path="/var/lib/kubelet/pods/bdd1a779-5e7b-45df-890c-b759b3b17f3c/volumes"
Nov 02 12:21:57.297933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.297900    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"]
Nov 02 12:21:57.298532 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.298516    4464 cpu_manager.go:414] "containerMap: removing container" podUID="aedbeb68-0cf8-48c4-a5e8-780c5db76876" containerName="revoke-keys"
Nov 02 12:21:57.298589 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.298535    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aedbeb68-0cf8-48c4-a5e8-780c5db76876" containerName="revoke-keys"
Nov 02 12:21:57.298695 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.298681    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="aedbeb68-0cf8-48c4-a5e8-780c5db76876" containerName="revoke-keys"
Nov 02 12:21:57.303644 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.303622    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"
Nov 02 12:21:57.357562 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.357529    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583a7279-0e1d-4542-8db7-946602fa9b37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17be0c70-9rn78\" (UID: \"583a7279-0e1d-4542-8db7-946602fa9b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"
Nov 02 12:21:57.357706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.357604    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb7md\" (UniqueName: \"kubernetes.io/projected/583a7279-0e1d-4542-8db7-946602fa9b37-kube-api-access-qb7md\") pod \"maas-api-revoke-sub-17be0c70-9rn78\" (UID: \"583a7279-0e1d-4542-8db7-946602fa9b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"
Nov 02 12:21:57.357785 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.357765    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583a7279-0e1d-4542-8db7-946602fa9b37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17be0c70-9rn78\" (UID: \"583a7279-0e1d-4542-8db7-946602fa9b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"
Nov 02 12:21:57.365699 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.365673    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"]
Nov 02 12:21:57.405117 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.405091    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:21:57.405251 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.405131    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:21:57.405251 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.405178    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:21:57.405251 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.405232    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:21:57.458859 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.458828    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583a7279-0e1d-4542-8db7-946602fa9b37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17be0c70-9rn78\" (UID: \"583a7279-0e1d-4542-8db7-946602fa9b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"
Nov 02 12:21:57.459035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.458926    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583a7279-0e1d-4542-8db7-946602fa9b37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17be0c70-9rn78\" (UID: \"583a7279-0e1d-4542-8db7-946602fa9b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"
Nov 02 12:21:57.459035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.458993    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb7md\" (UniqueName: \"kubernetes.io/projected/583a7279-0e1d-4542-8db7-946602fa9b37-kube-api-access-qb7md\") pod \"maas-api-revoke-sub-17be0c70-9rn78\" (UID: \"583a7279-0e1d-4542-8db7-946602fa9b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"
Nov 02 12:21:57.459696 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.459673    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583a7279-0e1d-4542-8db7-946602fa9b37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17be0c70-9rn78\" (UID: \"583a7279-0e1d-4542-8db7-946602fa9b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"
Nov 02 12:21:57.461380 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.461357    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583a7279-0e1d-4542-8db7-946602fa9b37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17be0c70-9rn78\" (UID: \"583a7279-0e1d-4542-8db7-946602fa9b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"
Nov 02 12:21:57.520781 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.520748    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb7md\" (UniqueName: \"kubernetes.io/projected/583a7279-0e1d-4542-8db7-946602fa9b37-kube-api-access-qb7md\") pod \"maas-api-revoke-sub-17be0c70-9rn78\" (UID: \"583a7279-0e1d-4542-8db7-946602fa9b37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"
Nov 02 12:21:57.617547 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.617467    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"
Nov 02 12:21:57.996028 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:57.996004    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"]
Nov 02 12:21:58.000657 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:21:58.000621    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583a7279_0e1d_4542_8db7_946602fa9b37.slice/crio-11a8f56a2fb2c6f6f14ed2551a0d2580b3d1cad53491f05b1ff2367da30efebc WatchSource:0}: Error finding container 11a8f56a2fb2c6f6f14ed2551a0d2580b3d1cad53491f05b1ff2367da30efebc: Status 404 returned error can't find the container with id 11a8f56a2fb2c6f6f14ed2551a0d2580b3d1cad53491f05b1ff2367da30efebc
Nov 02 12:21:58.269663 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:58.269634    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78" event={"ID":"583a7279-0e1d-4542-8db7-946602fa9b37","Type":"ContainerStarted","Data":"11a8f56a2fb2c6f6f14ed2551a0d2580b3d1cad53491f05b1ff2367da30efebc"}
Nov 02 12:21:58.291667 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:21:58.291635    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:21:59.279578 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:59.279543    4464 generic.go:356] "Generic (PLEG): container finished" podID="583a7279-0e1d-4542-8db7-946602fa9b37" containerID="e5d7c43ae91c1c0938090002bd4e167ef8f88ca461bfe4bca26d9c4f748fd471" exitCode=0
Nov 02 12:21:59.280027 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:21:59.279625    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78" event={"ID":"583a7279-0e1d-4542-8db7-946602fa9b37","Type":"ContainerDied","Data":"e5d7c43ae91c1c0938090002bd4e167ef8f88ca461bfe4bca26d9c4f748fd471"}
Nov 02 12:22:00.451329 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:00.451304    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"
Nov 02 12:22:00.590923 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:00.590855    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583a7279-0e1d-4542-8db7-946602fa9b37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583a7279-0e1d-4542-8db7-946602fa9b37-maas-api-openshift-ca\") pod \"583a7279-0e1d-4542-8db7-946602fa9b37\" (UID: \"583a7279-0e1d-4542-8db7-946602fa9b37\") "
Nov 02 12:22:00.591096 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:00.590924    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583a7279-0e1d-4542-8db7-946602fa9b37-kube-api-access-qb7md\" (UniqueName: \"kubernetes.io/projected/583a7279-0e1d-4542-8db7-946602fa9b37-kube-api-access-qb7md\") pod \"583a7279-0e1d-4542-8db7-946602fa9b37\" (UID: \"583a7279-0e1d-4542-8db7-946602fa9b37\") "
Nov 02 12:22:00.591096 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:00.590952    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583a7279-0e1d-4542-8db7-946602fa9b37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583a7279-0e1d-4542-8db7-946602fa9b37-maas-api-xks-ca\") pod \"583a7279-0e1d-4542-8db7-946602fa9b37\" (UID: \"583a7279-0e1d-4542-8db7-946602fa9b37\") "
Nov 02 12:22:00.591305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:00.591281    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583a7279-0e1d-4542-8db7-946602fa9b37-maas-api-openshift-ca" pod "583a7279-0e1d-4542-8db7-946602fa9b37" (UID: "583a7279-0e1d-4542-8db7-946602fa9b37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:22:00.591559 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:00.591529    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583a7279-0e1d-4542-8db7-946602fa9b37-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:00.593087 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:00.593059    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583a7279-0e1d-4542-8db7-946602fa9b37-maas-api-xks-ca" pod "583a7279-0e1d-4542-8db7-946602fa9b37" (UID: "583a7279-0e1d-4542-8db7-946602fa9b37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:22:00.593087 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:00.593073    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583a7279-0e1d-4542-8db7-946602fa9b37-kube-api-access-qb7md" pod "583a7279-0e1d-4542-8db7-946602fa9b37" (UID: "583a7279-0e1d-4542-8db7-946602fa9b37"). InnerVolumeSpecName "kube-api-access-qb7md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:22:00.692582 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:00.692554    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb7md\" (UniqueName: \"kubernetes.io/projected/583a7279-0e1d-4542-8db7-946602fa9b37-kube-api-access-qb7md\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:00.692582 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:00.692579    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583a7279-0e1d-4542-8db7-946602fa9b37-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:01.296856 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:01.296828    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"
Nov 02 12:22:01.296856 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:01.296833    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78" event={"ID":"583a7279-0e1d-4542-8db7-946602fa9b37","Type":"ContainerDied","Data":"11a8f56a2fb2c6f6f14ed2551a0d2580b3d1cad53491f05b1ff2367da30efebc"}
Nov 02 12:22:01.297086 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:01.296878    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11a8f56a2fb2c6f6f14ed2551a0d2580b3d1cad53491f05b1ff2367da30efebc"
Nov 02 12:22:10.050459 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:10.050427    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"]
Nov 02 12:22:10.100918 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:10.100886    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba9b8cf1-htvhp"]
Nov 02 12:22:11.582560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:11.582532    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f8c8665-1f21-4d88-9119-6d4826b40a8d" path="/var/lib/kubelet/pods/0f8c8665-1f21-4d88-9119-6d4826b40a8d/volumes"
Nov 02 12:22:12.345037 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.345004    4464 scope.go:122] "RemoveContainer" containerID="0876770036ee8d7b5d2bb6671b1348bdb75e47e98d0904500e7014404cf1eeb9"
Nov 02 12:22:12.355346 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.355327    4464 scope.go:122] "RemoveContainer" containerID="512d6d81a5938e5d5f01e4a16cf237d2572519e616226ef765b6853479c1cfe7"
Nov 02 12:22:12.365582 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.365541    4464 scope.go:122] "RemoveContainer" containerID="22fb6b054c6101444093e3017b5d217e01111fc0e9b801de6b3991f564b07421"
Nov 02 12:22:12.375566 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.375546    4464 scope.go:122] "RemoveContainer" containerID="0ce6d9cec5c37c73401231e0f7b871bc143237a84bdc30bbddef817051f010c6"
Nov 02 12:22:12.384038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.384012    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"]
Nov 02 12:22:12.384697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.384676    4464 cpu_manager.go:414] "containerMap: removing container" podUID="583a7279-0e1d-4542-8db7-946602fa9b37" containerName="revoke-keys"
Nov 02 12:22:12.384786 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.384698    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583a7279-0e1d-4542-8db7-946602fa9b37" containerName="revoke-keys"
Nov 02 12:22:12.384887 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.384873    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="583a7279-0e1d-4542-8db7-946602fa9b37" containerName="revoke-keys"
Nov 02 12:22:12.385845 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.385822    4464 scope.go:122] "RemoveContainer" containerID="aee803f7766dac33c3bdc57e6dbe5b463042bfdebd45476b916d6b8fdfc1a577"
Nov 02 12:22:12.390485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.390467    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"
Nov 02 12:22:12.399750 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.399728    4464 scope.go:122] "RemoveContainer" containerID="0b8a5418ee59300f0cdcbf8f48a0e19f7da73c260ccc5239b977a03d0c2aa5fb"
Nov 02 12:22:12.410662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.410645    4464 scope.go:122] "RemoveContainer" containerID="80ff7eb703ee2544a07fc7e7ac86ef185ebaf035b68a651713e320e94cfc059e"
Nov 02 12:22:12.420812 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.420680    4464 scope.go:122] "RemoveContainer" containerID="7c353570839f47f9063df7ed62593f02e20d5880870b84afddc8e76f6fcb4aa8"
Nov 02 12:22:12.431021 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.430997    4464 scope.go:122] "RemoveContainer" containerID="11dac207c29ddeea5d0be147e68d4b85da562c97d41eb411642095f1bcd6a373"
Nov 02 12:22:12.440260 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.440240    4464 scope.go:122] "RemoveContainer" containerID="051b60ce4f07c18d05bec284c9f7aa5f2003401b39c27092849820a5d4d031fe"
Nov 02 12:22:12.445897 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.445875    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:22:12.449538 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.449513    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"]
Nov 02 12:22:12.480082 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.479928    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363e0725-226b-4e32-996c-7023534370dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5\" (UID: \"363e0725-226b-4e32-996c-7023534370dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"
Nov 02 12:22:12.480082 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.479989    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sxl7\" (UniqueName: \"kubernetes.io/projected/363e0725-226b-4e32-996c-7023534370dd-kube-api-access-6sxl7\") pod \"maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5\" (UID: \"363e0725-226b-4e32-996c-7023534370dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"
Nov 02 12:22:12.480293 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.480203    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363e0725-226b-4e32-996c-7023534370dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5\" (UID: \"363e0725-226b-4e32-996c-7023534370dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"
Nov 02 12:22:12.495192 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.495170    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:22:12.495496 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.495174    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:22:12.495496 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.495176    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:22:12.581964 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.581923    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363e0725-226b-4e32-996c-7023534370dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5\" (UID: \"363e0725-226b-4e32-996c-7023534370dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"
Nov 02 12:22:12.582177 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.582119    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363e0725-226b-4e32-996c-7023534370dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5\" (UID: \"363e0725-226b-4e32-996c-7023534370dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"
Nov 02 12:22:12.582177 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.582151    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sxl7\" (UniqueName: \"kubernetes.io/projected/363e0725-226b-4e32-996c-7023534370dd-kube-api-access-6sxl7\") pod \"maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5\" (UID: \"363e0725-226b-4e32-996c-7023534370dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"
Nov 02 12:22:12.582786 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.582762    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363e0725-226b-4e32-996c-7023534370dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5\" (UID: \"363e0725-226b-4e32-996c-7023534370dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"
Nov 02 12:22:12.584527 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.584505    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363e0725-226b-4e32-996c-7023534370dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5\" (UID: \"363e0725-226b-4e32-996c-7023534370dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"
Nov 02 12:22:12.644328 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.644261    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sxl7\" (UniqueName: \"kubernetes.io/projected/363e0725-226b-4e32-996c-7023534370dd-kube-api-access-6sxl7\") pod \"maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5\" (UID: \"363e0725-226b-4e32-996c-7023534370dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"
Nov 02 12:22:12.710432 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:12.710396    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"
Nov 02 12:22:13.090757 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:22:13.090723    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363e0725_226b_4e32_996c_7023534370dd.slice/crio-d8c9d0109a3613f96c94bc226563659e7c6b025fa206276ab75db1e6cfb27902 WatchSource:0}: Error finding container d8c9d0109a3613f96c94bc226563659e7c6b025fa206276ab75db1e6cfb27902: Status 404 returned error can't find the container with id d8c9d0109a3613f96c94bc226563659e7c6b025fa206276ab75db1e6cfb27902
Nov 02 12:22:13.093708 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:13.093683    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"]
Nov 02 12:22:13.401933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:13.401893    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5" event={"ID":"363e0725-226b-4e32-996c-7023534370dd","Type":"ContainerStarted","Data":"dbf5f4c6f02931a22852ddb4b7898054177c790d5e66ef957732ea562041cd0e"}
Nov 02 12:22:13.402115 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:13.401958    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5" event={"ID":"363e0725-226b-4e32-996c-7023534370dd","Type":"ContainerStarted","Data":"d8c9d0109a3613f96c94bc226563659e7c6b025fa206276ab75db1e6cfb27902"}
Nov 02 12:22:13.520761 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:13.520713    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5" podStartSLOduration=1.5206987189999999 podStartE2EDuration="1.520698719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:22:13.51826888 +0000 UTC m=+1562.081977292" watchObservedRunningTime="2026-11-02 12:22:13.520698719 +0000 UTC m=+1562.084407127"
Nov 02 12:22:14.411735 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:14.411703    4464 generic.go:356] "Generic (PLEG): container finished" podID="363e0725-226b-4e32-996c-7023534370dd" containerID="dbf5f4c6f02931a22852ddb4b7898054177c790d5e66ef957732ea562041cd0e" exitCode=0
Nov 02 12:22:14.412151 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:14.411788    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5" event={"ID":"363e0725-226b-4e32-996c-7023534370dd","Type":"ContainerDied","Data":"dbf5f4c6f02931a22852ddb4b7898054177c790d5e66ef957732ea562041cd0e"}
Nov 02 12:22:15.242917 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.242878    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"]
Nov 02 12:22:15.247195 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.247173    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"
Nov 02 12:22:15.306620 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.306579    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"]
Nov 02 12:22:15.311439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.311404    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b8494e-hj2xj\" (UID: \"8c2aeb0d-0494-4743-8b75-dfe3cfb568af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"
Nov 02 12:22:15.311439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.311450    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b8494e-hj2xj\" (UID: \"8c2aeb0d-0494-4743-8b75-dfe3cfb568af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"
Nov 02 12:22:15.311937 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.311827    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84rdm\" (UniqueName: \"kubernetes.io/projected/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-kube-api-access-84rdm\") pod \"maas-api-revoke-sub-a2b8494e-hj2xj\" (UID: \"8c2aeb0d-0494-4743-8b75-dfe3cfb568af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"
Nov 02 12:22:15.412716 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.412678    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b8494e-hj2xj\" (UID: \"8c2aeb0d-0494-4743-8b75-dfe3cfb568af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"
Nov 02 12:22:15.413186 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.412723    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b8494e-hj2xj\" (UID: \"8c2aeb0d-0494-4743-8b75-dfe3cfb568af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"
Nov 02 12:22:15.413186 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.412865    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84rdm\" (UniqueName: \"kubernetes.io/projected/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-kube-api-access-84rdm\") pod \"maas-api-revoke-sub-a2b8494e-hj2xj\" (UID: \"8c2aeb0d-0494-4743-8b75-dfe3cfb568af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"
Nov 02 12:22:15.413504 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.413481    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b8494e-hj2xj\" (UID: \"8c2aeb0d-0494-4743-8b75-dfe3cfb568af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"
Nov 02 12:22:15.415186 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.415161    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b8494e-hj2xj\" (UID: \"8c2aeb0d-0494-4743-8b75-dfe3cfb568af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"
Nov 02 12:22:15.474548 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.474517    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84rdm\" (UniqueName: \"kubernetes.io/projected/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-kube-api-access-84rdm\") pod \"maas-api-revoke-sub-a2b8494e-hj2xj\" (UID: \"8c2aeb0d-0494-4743-8b75-dfe3cfb568af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"
Nov 02 12:22:15.560400 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.560363    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"
Nov 02 12:22:15.592217 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.592189    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"
Nov 02 12:22:15.615616 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.615584    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/363e0725-226b-4e32-996c-7023534370dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363e0725-226b-4e32-996c-7023534370dd-maas-api-xks-ca\") pod \"363e0725-226b-4e32-996c-7023534370dd\" (UID: \"363e0725-226b-4e32-996c-7023534370dd\") "
Nov 02 12:22:15.615789 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.615642    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363e0725-226b-4e32-996c-7023534370dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363e0725-226b-4e32-996c-7023534370dd-maas-api-openshift-ca\") pod \"363e0725-226b-4e32-996c-7023534370dd\" (UID: \"363e0725-226b-4e32-996c-7023534370dd\") "
Nov 02 12:22:15.615789 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.615666    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363e0725-226b-4e32-996c-7023534370dd-kube-api-access-6sxl7\" (UniqueName: \"kubernetes.io/projected/363e0725-226b-4e32-996c-7023534370dd-kube-api-access-6sxl7\") pod \"363e0725-226b-4e32-996c-7023534370dd\" (UID: \"363e0725-226b-4e32-996c-7023534370dd\") "
Nov 02 12:22:15.616354 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.616099    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363e0725-226b-4e32-996c-7023534370dd-maas-api-openshift-ca" pod "363e0725-226b-4e32-996c-7023534370dd" (UID: "363e0725-226b-4e32-996c-7023534370dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:22:15.617029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.616999    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363e0725-226b-4e32-996c-7023534370dd-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:15.622636 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.622608    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363e0725-226b-4e32-996c-7023534370dd-kube-api-access-6sxl7" pod "363e0725-226b-4e32-996c-7023534370dd" (UID: "363e0725-226b-4e32-996c-7023534370dd"). InnerVolumeSpecName "kube-api-access-6sxl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:22:15.643071 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.643034    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363e0725-226b-4e32-996c-7023534370dd-maas-api-xks-ca" pod "363e0725-226b-4e32-996c-7023534370dd" (UID: "363e0725-226b-4e32-996c-7023534370dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:22:15.719076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.719029    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363e0725-226b-4e32-996c-7023534370dd-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:15.719076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.719075    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sxl7\" (UniqueName: \"kubernetes.io/projected/363e0725-226b-4e32-996c-7023534370dd-kube-api-access-6sxl7\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:15.946899 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:22:15.946868    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2aeb0d_0494_4743_8b75_dfe3cfb568af.slice/crio-efdee0a49daf173a1dc8a609c8c501bbb1e067148de7eee9bfc0536843d4cb16 WatchSource:0}: Error finding container efdee0a49daf173a1dc8a609c8c501bbb1e067148de7eee9bfc0536843d4cb16: Status 404 returned error can't find the container with id efdee0a49daf173a1dc8a609c8c501bbb1e067148de7eee9bfc0536843d4cb16
Nov 02 12:22:15.947561 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:15.947535    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"]
Nov 02 12:22:16.429805 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:16.429774    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"
Nov 02 12:22:16.429805 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:16.429779    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5" event={"ID":"363e0725-226b-4e32-996c-7023534370dd","Type":"ContainerDied","Data":"d8c9d0109a3613f96c94bc226563659e7c6b025fa206276ab75db1e6cfb27902"}
Nov 02 12:22:16.430356 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:16.429840    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8c9d0109a3613f96c94bc226563659e7c6b025fa206276ab75db1e6cfb27902"
Nov 02 12:22:16.433051 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:16.432384    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj" event={"ID":"8c2aeb0d-0494-4743-8b75-dfe3cfb568af","Type":"ContainerStarted","Data":"219c36379084d5b6521dd43fdbdadf7c81fad38f9eebbac2a473f6a5bcd53acd"}
Nov 02 12:22:16.433051 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:16.432442    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj" event={"ID":"8c2aeb0d-0494-4743-8b75-dfe3cfb568af","Type":"ContainerStarted","Data":"efdee0a49daf173a1dc8a609c8c501bbb1e067148de7eee9bfc0536843d4cb16"}
Nov 02 12:22:16.553463 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:16.553364    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj" podStartSLOduration=1.553351025 podStartE2EDuration="1.553351025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:22:16.549882286 +0000 UTC m=+1565.113590722" watchObservedRunningTime="2026-11-02 12:22:16.553351025 +0000 UTC m=+1565.117059438"
Nov 02 12:22:17.441278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:17.441242    4464 generic.go:356] "Generic (PLEG): container finished" podID="8c2aeb0d-0494-4743-8b75-dfe3cfb568af" containerID="219c36379084d5b6521dd43fdbdadf7c81fad38f9eebbac2a473f6a5bcd53acd" exitCode=0
Nov 02 12:22:17.441656 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:17.441322    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj" event={"ID":"8c2aeb0d-0494-4743-8b75-dfe3cfb568af","Type":"ContainerDied","Data":"219c36379084d5b6521dd43fdbdadf7c81fad38f9eebbac2a473f6a5bcd53acd"}
Nov 02 12:22:18.601288 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:18.601264    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"
Nov 02 12:22:18.649690 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:18.649658    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-kube-api-access-84rdm\" (UniqueName: \"kubernetes.io/projected/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-kube-api-access-84rdm\") pod \"8c2aeb0d-0494-4743-8b75-dfe3cfb568af\" (UID: \"8c2aeb0d-0494-4743-8b75-dfe3cfb568af\") "
Nov 02 12:22:18.649849 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:18.649760    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-maas-api-xks-ca\") pod \"8c2aeb0d-0494-4743-8b75-dfe3cfb568af\" (UID: \"8c2aeb0d-0494-4743-8b75-dfe3cfb568af\") "
Nov 02 12:22:18.649892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:18.649868    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-maas-api-openshift-ca\") pod \"8c2aeb0d-0494-4743-8b75-dfe3cfb568af\" (UID: \"8c2aeb0d-0494-4743-8b75-dfe3cfb568af\") "
Nov 02 12:22:18.650535 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:18.650205    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-maas-api-openshift-ca" pod "8c2aeb0d-0494-4743-8b75-dfe3cfb568af" (UID: "8c2aeb0d-0494-4743-8b75-dfe3cfb568af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:22:18.650696 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:18.650676    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:18.651918 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:18.651897    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-maas-api-xks-ca" pod "8c2aeb0d-0494-4743-8b75-dfe3cfb568af" (UID: "8c2aeb0d-0494-4743-8b75-dfe3cfb568af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:22:18.652020 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:18.651902    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-kube-api-access-84rdm" pod "8c2aeb0d-0494-4743-8b75-dfe3cfb568af" (UID: "8c2aeb0d-0494-4743-8b75-dfe3cfb568af"). InnerVolumeSpecName "kube-api-access-84rdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:22:18.751405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:18.751359    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84rdm\" (UniqueName: \"kubernetes.io/projected/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-kube-api-access-84rdm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:18.751405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:18.751400    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c2aeb0d-0494-4743-8b75-dfe3cfb568af-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:19.076616 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:19.073396    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:22:19.456924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:19.456842    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"
Nov 02 12:22:19.456924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:19.456871    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj" event={"ID":"8c2aeb0d-0494-4743-8b75-dfe3cfb568af","Type":"ContainerDied","Data":"efdee0a49daf173a1dc8a609c8c501bbb1e067148de7eee9bfc0536843d4cb16"}
Nov 02 12:22:19.456924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:19.456918    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efdee0a49daf173a1dc8a609c8c501bbb1e067148de7eee9bfc0536843d4cb16"
Nov 02 12:22:22.230161 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:22.230116    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:22:22.230621 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:22.230208    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p" podUID="dda61e72-8ab1-4390-8295-83a8fc113578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:22:23.998699 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:23.998664    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"]
Nov 02 12:22:24.113778 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:24.113743    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"]
Nov 02 12:22:26.048384 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:26.048309    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"]
Nov 02 12:22:26.098703 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:26.098665    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-265722de-ff6lg"]
Nov 02 12:22:27.582616 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:27.582584    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5223feb-e5f0-4e64-95a9-c7f4235e99b2" path="/var/lib/kubelet/pods/c5223feb-e5f0-4e64-95a9-c7f4235e99b2/volumes"
Nov 02 12:22:28.292257 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:22:28.292216    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:22:28.327073 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.327043    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"]
Nov 02 12:22:28.327682 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.327668    4464 cpu_manager.go:414] "containerMap: removing container" podUID="8c2aeb0d-0494-4743-8b75-dfe3cfb568af" containerName="revoke-keys"
Nov 02 12:22:28.327726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.327684    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c2aeb0d-0494-4743-8b75-dfe3cfb568af" containerName="revoke-keys"
Nov 02 12:22:28.327726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.327708    4464 cpu_manager.go:414] "containerMap: removing container" podUID="363e0725-226b-4e32-996c-7023534370dd" containerName="revoke-keys"
Nov 02 12:22:28.327726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.327713    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363e0725-226b-4e32-996c-7023534370dd" containerName="revoke-keys"
Nov 02 12:22:28.327830 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.327804    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c2aeb0d-0494-4743-8b75-dfe3cfb568af" containerName="revoke-keys"
Nov 02 12:22:28.327830 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.327818    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="363e0725-226b-4e32-996c-7023534370dd" containerName="revoke-keys"
Nov 02 12:22:28.332776 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.332756    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"
Nov 02 12:22:28.389804 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.389774    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"]
Nov 02 12:22:28.435243 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.435210    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:22:28.435243 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.435213    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:22:28.435243 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.435223    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:22:28.435470 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.435270    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:22:28.445912 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.445890    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cace0f3d-c048-4f89-9fb0-180afb8b468c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-352a61b2-rjfpn\" (UID: \"cace0f3d-c048-4f89-9fb0-180afb8b468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"
Nov 02 12:22:28.446036 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.445968    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cljmn\" (UniqueName: \"kubernetes.io/projected/cace0f3d-c048-4f89-9fb0-180afb8b468c-kube-api-access-cljmn\") pod \"maas-api-revoke-sub-352a61b2-rjfpn\" (UID: \"cace0f3d-c048-4f89-9fb0-180afb8b468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"
Nov 02 12:22:28.446102 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.446083    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cace0f3d-c048-4f89-9fb0-180afb8b468c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-352a61b2-rjfpn\" (UID: \"cace0f3d-c048-4f89-9fb0-180afb8b468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"
Nov 02 12:22:28.547076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.547003    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cace0f3d-c048-4f89-9fb0-180afb8b468c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-352a61b2-rjfpn\" (UID: \"cace0f3d-c048-4f89-9fb0-180afb8b468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"
Nov 02 12:22:28.547230 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.547082    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cljmn\" (UniqueName: \"kubernetes.io/projected/cace0f3d-c048-4f89-9fb0-180afb8b468c-kube-api-access-cljmn\") pod \"maas-api-revoke-sub-352a61b2-rjfpn\" (UID: \"cace0f3d-c048-4f89-9fb0-180afb8b468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"
Nov 02 12:22:28.547296 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.547244    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cace0f3d-c048-4f89-9fb0-180afb8b468c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-352a61b2-rjfpn\" (UID: \"cace0f3d-c048-4f89-9fb0-180afb8b468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"
Nov 02 12:22:28.547825 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.547792    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cace0f3d-c048-4f89-9fb0-180afb8b468c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-352a61b2-rjfpn\" (UID: \"cace0f3d-c048-4f89-9fb0-180afb8b468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"
Nov 02 12:22:28.549501 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.549477    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cace0f3d-c048-4f89-9fb0-180afb8b468c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-352a61b2-rjfpn\" (UID: \"cace0f3d-c048-4f89-9fb0-180afb8b468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"
Nov 02 12:22:28.609165 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.609136    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cljmn\" (UniqueName: \"kubernetes.io/projected/cace0f3d-c048-4f89-9fb0-180afb8b468c-kube-api-access-cljmn\") pod \"maas-api-revoke-sub-352a61b2-rjfpn\" (UID: \"cace0f3d-c048-4f89-9fb0-180afb8b468c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"
Nov 02 12:22:28.648064 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.648036    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"
Nov 02 12:22:28.999823 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:28.999781    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2" podUID="8897bb5d-ac09-4389-a3fe-65bae125f6be" containerName="payload-processing" containerID="cri-o://525de643b88494809697cc356e287e325a2b6f22ae51d517b1eaa694dc5183ec" gracePeriod=25
Nov 02 12:22:29.025437 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.025410    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"]
Nov 02 12:22:29.081922 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:22:29.081893    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcace0f3d_c048_4f89_9fb0_180afb8b468c.slice/crio-8bdbcf77b09e417b1ef57b5d4d080e9121f79ec9e933e66d502c60f8ba5a4aa1 WatchSource:0}: Error finding container 8bdbcf77b09e417b1ef57b5d4d080e9121f79ec9e933e66d502c60f8ba5a4aa1: Status 404 returned error can't find the container with id 8bdbcf77b09e417b1ef57b5d4d080e9121f79ec9e933e66d502c60f8ba5a4aa1
Nov 02 12:22:29.114414 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.114365    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt" podUID="9c2a8ed4-5986-40b4-a810-452e01712e38" containerName="payload-pre-processing" containerID="cri-o://1c6e9a3f63401fe79beb17ada2dce13e6fe1961baf65b0c55e7764b37b0055a5" gracePeriod=25
Nov 02 12:22:29.240150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.240117    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"
Nov 02 12:22:29.356580 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.356549    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8897bb5d-ac09-4389-a3fe-65bae125f6be-kube-api-access-l4khq\" (UniqueName: \"kubernetes.io/projected/8897bb5d-ac09-4389-a3fe-65bae125f6be-kube-api-access-l4khq\") pod \"8897bb5d-ac09-4389-a3fe-65bae125f6be\" (UID: \"8897bb5d-ac09-4389-a3fe-65bae125f6be\") "
Nov 02 12:22:29.356747 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.356669    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8897bb5d-ac09-4389-a3fe-65bae125f6be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8897bb5d-ac09-4389-a3fe-65bae125f6be-plugins-config-volume\") pod \"8897bb5d-ac09-4389-a3fe-65bae125f6be\" (UID: \"8897bb5d-ac09-4389-a3fe-65bae125f6be\") "
Nov 02 12:22:29.357311 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.357116    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8897bb5d-ac09-4389-a3fe-65bae125f6be-plugins-config-volume" pod "8897bb5d-ac09-4389-a3fe-65bae125f6be" (UID: "8897bb5d-ac09-4389-a3fe-65bae125f6be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:22:29.357311 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.357240    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8897bb5d-ac09-4389-a3fe-65bae125f6be-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:29.360395 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.360367    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8897bb5d-ac09-4389-a3fe-65bae125f6be-kube-api-access-l4khq" pod "8897bb5d-ac09-4389-a3fe-65bae125f6be" (UID: "8897bb5d-ac09-4389-a3fe-65bae125f6be"). InnerVolumeSpecName "kube-api-access-l4khq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:22:29.374372 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.374349    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"
Nov 02 12:22:29.458207 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.458172    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c2a8ed4-5986-40b4-a810-452e01712e38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2a8ed4-5986-40b4-a810-452e01712e38-plugins-config-volume\") pod \"9c2a8ed4-5986-40b4-a810-452e01712e38\" (UID: \"9c2a8ed4-5986-40b4-a810-452e01712e38\") "
Nov 02 12:22:29.458207 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.458206    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c2a8ed4-5986-40b4-a810-452e01712e38-kube-api-access-57r2l\" (UniqueName: \"kubernetes.io/projected/9c2a8ed4-5986-40b4-a810-452e01712e38-kube-api-access-57r2l\") pod \"9c2a8ed4-5986-40b4-a810-452e01712e38\" (UID: \"9c2a8ed4-5986-40b4-a810-452e01712e38\") "
Nov 02 12:22:29.458601 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.458579    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2a8ed4-5986-40b4-a810-452e01712e38-plugins-config-volume" pod "9c2a8ed4-5986-40b4-a810-452e01712e38" (UID: "9c2a8ed4-5986-40b4-a810-452e01712e38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:22:29.458767 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.458751    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4khq\" (UniqueName: \"kubernetes.io/projected/8897bb5d-ac09-4389-a3fe-65bae125f6be-kube-api-access-l4khq\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:29.458838 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.458772    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2a8ed4-5986-40b4-a810-452e01712e38-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:29.460368 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.460345    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2a8ed4-5986-40b4-a810-452e01712e38-kube-api-access-57r2l" pod "9c2a8ed4-5986-40b4-a810-452e01712e38" (UID: "9c2a8ed4-5986-40b4-a810-452e01712e38"). InnerVolumeSpecName "kube-api-access-57r2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:22:29.560253 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.560223    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57r2l\" (UniqueName: \"kubernetes.io/projected/9c2a8ed4-5986-40b4-a810-452e01712e38-kube-api-access-57r2l\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:29.585825 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.585791    4464 generic.go:356] "Generic (PLEG): container finished" podID="9c2a8ed4-5986-40b4-a810-452e01712e38" containerID="1c6e9a3f63401fe79beb17ada2dce13e6fe1961baf65b0c55e7764b37b0055a5" exitCode=0
Nov 02 12:22:29.585952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.585827    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"
Nov 02 12:22:29.585952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.585873    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt" event={"ID":"9c2a8ed4-5986-40b4-a810-452e01712e38","Type":"ContainerDied","Data":"1c6e9a3f63401fe79beb17ada2dce13e6fe1961baf65b0c55e7764b37b0055a5"}
Nov 02 12:22:29.585952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.585929    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt" event={"ID":"9c2a8ed4-5986-40b4-a810-452e01712e38","Type":"ContainerDied","Data":"c0ba11e2f3a788423caf74bee955faf9c2187cebafe3c5990a1497b3be1254e1"}
Nov 02 12:22:29.586143 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.585958    4464 scope.go:122] "RemoveContainer" containerID="1c6e9a3f63401fe79beb17ada2dce13e6fe1961baf65b0c55e7764b37b0055a5"
Nov 02 12:22:29.587853 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.587823    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn" event={"ID":"cace0f3d-c048-4f89-9fb0-180afb8b468c","Type":"ContainerStarted","Data":"2b23e24ae17c5399f4cad4facf9bed5b34119f87a45a01d25b2a69d6618165de"}
Nov 02 12:22:29.587960 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.587878    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn" event={"ID":"cace0f3d-c048-4f89-9fb0-180afb8b468c","Type":"ContainerStarted","Data":"8bdbcf77b09e417b1ef57b5d4d080e9121f79ec9e933e66d502c60f8ba5a4aa1"}
Nov 02 12:22:29.589932 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.589909    4464 generic.go:356] "Generic (PLEG): container finished" podID="8897bb5d-ac09-4389-a3fe-65bae125f6be" containerID="525de643b88494809697cc356e287e325a2b6f22ae51d517b1eaa694dc5183ec" exitCode=0
Nov 02 12:22:29.590063 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.589954    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"
Nov 02 12:22:29.590063 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.590029    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2" event={"ID":"8897bb5d-ac09-4389-a3fe-65bae125f6be","Type":"ContainerDied","Data":"525de643b88494809697cc356e287e325a2b6f22ae51d517b1eaa694dc5183ec"}
Nov 02 12:22:29.590188 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.590088    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2" event={"ID":"8897bb5d-ac09-4389-a3fe-65bae125f6be","Type":"ContainerDied","Data":"041ee247d9e86b9c815a732c0ece01ca325f706db7008b10052eb3960761b29f"}
Nov 02 12:22:29.597537 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.597522    4464 scope.go:122] "RemoveContainer" containerID="1c6e9a3f63401fe79beb17ada2dce13e6fe1961baf65b0c55e7764b37b0055a5"
Nov 02 12:22:29.597902 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:22:29.597879    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6e9a3f63401fe79beb17ada2dce13e6fe1961baf65b0c55e7764b37b0055a5\": container with ID starting with 1c6e9a3f63401fe79beb17ada2dce13e6fe1961baf65b0c55e7764b37b0055a5 not found: ID does not exist" containerID="1c6e9a3f63401fe79beb17ada2dce13e6fe1961baf65b0c55e7764b37b0055a5"
Nov 02 12:22:29.598003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.597907    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6e9a3f63401fe79beb17ada2dce13e6fe1961baf65b0c55e7764b37b0055a5"} err="failed to get container status \"1c6e9a3f63401fe79beb17ada2dce13e6fe1961baf65b0c55e7764b37b0055a5\": rpc error: code = NotFound desc = could not find container \"1c6e9a3f63401fe79beb17ada2dce13e6fe1961baf65b0c55e7764b37b0055a5\": container with ID starting with 1c6e9a3f63401fe79beb17ada2dce13e6fe1961baf65b0c55e7764b37b0055a5 not found: ID does not exist"
Nov 02 12:22:29.598003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.597929    4464 scope.go:122] "RemoveContainer" containerID="525de643b88494809697cc356e287e325a2b6f22ae51d517b1eaa694dc5183ec"
Nov 02 12:22:29.607537 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.607522    4464 scope.go:122] "RemoveContainer" containerID="525de643b88494809697cc356e287e325a2b6f22ae51d517b1eaa694dc5183ec"
Nov 02 12:22:29.607788 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:22:29.607769    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525de643b88494809697cc356e287e325a2b6f22ae51d517b1eaa694dc5183ec\": container with ID starting with 525de643b88494809697cc356e287e325a2b6f22ae51d517b1eaa694dc5183ec not found: ID does not exist" containerID="525de643b88494809697cc356e287e325a2b6f22ae51d517b1eaa694dc5183ec"
Nov 02 12:22:29.607840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.607793    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525de643b88494809697cc356e287e325a2b6f22ae51d517b1eaa694dc5183ec"} err="failed to get container status \"525de643b88494809697cc356e287e325a2b6f22ae51d517b1eaa694dc5183ec\": rpc error: code = NotFound desc = could not find container \"525de643b88494809697cc356e287e325a2b6f22ae51d517b1eaa694dc5183ec\": container with ID starting with 525de643b88494809697cc356e287e325a2b6f22ae51d517b1eaa694dc5183ec not found: ID does not exist"
Nov 02 12:22:29.755542 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.755503    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"]
Nov 02 12:22:29.760652 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.760624    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fbf1444a-56db895c4c-5vjxt"]
Nov 02 12:22:29.871577 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.871493    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"]
Nov 02 12:22:29.875578 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.875549    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fbf1444a-5b8dbb7699-rtgv2"]
Nov 02 12:22:29.989149 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:29.989090    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn" podStartSLOduration=1.989072732 podStartE2EDuration="1.989072732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:22:29.988515441 +0000 UTC m=+1578.552223851" watchObservedRunningTime="2026-11-02 12:22:29.989072732 +0000 UTC m=+1578.552781143"
Nov 02 12:22:30.599542 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.599504    4464 generic.go:356] "Generic (PLEG): container finished" podID="cace0f3d-c048-4f89-9fb0-180afb8b468c" containerID="2b23e24ae17c5399f4cad4facf9bed5b34119f87a45a01d25b2a69d6618165de" exitCode=0
Nov 02 12:22:30.599725 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.599579    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn" event={"ID":"cace0f3d-c048-4f89-9fb0-180afb8b468c","Type":"ContainerDied","Data":"2b23e24ae17c5399f4cad4facf9bed5b34119f87a45a01d25b2a69d6618165de"}
Nov 02 12:22:30.708411 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.708378    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"]
Nov 02 12:22:30.709013 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.708987    4464 cpu_manager.go:414] "containerMap: removing container" podUID="9c2a8ed4-5986-40b4-a810-452e01712e38" containerName="payload-pre-processing"
Nov 02 12:22:30.709013 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.709016    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c2a8ed4-5986-40b4-a810-452e01712e38" containerName="payload-pre-processing"
Nov 02 12:22:30.709114 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.709051    4464 cpu_manager.go:414] "containerMap: removing container" podUID="8897bb5d-ac09-4389-a3fe-65bae125f6be" containerName="payload-processing"
Nov 02 12:22:30.709114 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.709056    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8897bb5d-ac09-4389-a3fe-65bae125f6be" containerName="payload-processing"
Nov 02 12:22:30.709172 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.709142    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c2a8ed4-5986-40b4-a810-452e01712e38" containerName="payload-pre-processing"
Nov 02 12:22:30.709172 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.709156    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="8897bb5d-ac09-4389-a3fe-65bae125f6be" containerName="payload-processing"
Nov 02 12:22:30.712641 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.712624    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"
Nov 02 12:22:30.738487 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.738461    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"]
Nov 02 12:22:30.774846 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.774802    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdzcn\" (UniqueName: \"kubernetes.io/projected/2b4233ad-7e42-44ee-b826-6106d6278ee7-kube-api-access-mdzcn\") pod \"maas-api-revoke-sub-4445878c-kqmcm\" (UID: \"2b4233ad-7e42-44ee-b826-6106d6278ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"
Nov 02 12:22:30.775316 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.774864    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b4233ad-7e42-44ee-b826-6106d6278ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4445878c-kqmcm\" (UID: \"2b4233ad-7e42-44ee-b826-6106d6278ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"
Nov 02 12:22:30.775316 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.775105    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b4233ad-7e42-44ee-b826-6106d6278ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4445878c-kqmcm\" (UID: \"2b4233ad-7e42-44ee-b826-6106d6278ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"
Nov 02 12:22:30.876781 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.876705    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b4233ad-7e42-44ee-b826-6106d6278ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4445878c-kqmcm\" (UID: \"2b4233ad-7e42-44ee-b826-6106d6278ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"
Nov 02 12:22:30.876933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.876802    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdzcn\" (UniqueName: \"kubernetes.io/projected/2b4233ad-7e42-44ee-b826-6106d6278ee7-kube-api-access-mdzcn\") pod \"maas-api-revoke-sub-4445878c-kqmcm\" (UID: \"2b4233ad-7e42-44ee-b826-6106d6278ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"
Nov 02 12:22:30.876933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.876826    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b4233ad-7e42-44ee-b826-6106d6278ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4445878c-kqmcm\" (UID: \"2b4233ad-7e42-44ee-b826-6106d6278ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"
Nov 02 12:22:30.877461 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.877433    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b4233ad-7e42-44ee-b826-6106d6278ee7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4445878c-kqmcm\" (UID: \"2b4233ad-7e42-44ee-b826-6106d6278ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"
Nov 02 12:22:30.879152 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.879135    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b4233ad-7e42-44ee-b826-6106d6278ee7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4445878c-kqmcm\" (UID: \"2b4233ad-7e42-44ee-b826-6106d6278ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"
Nov 02 12:22:30.938893 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:30.938863    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdzcn\" (UniqueName: \"kubernetes.io/projected/2b4233ad-7e42-44ee-b826-6106d6278ee7-kube-api-access-mdzcn\") pod \"maas-api-revoke-sub-4445878c-kqmcm\" (UID: \"2b4233ad-7e42-44ee-b826-6106d6278ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"
Nov 02 12:22:31.026817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:31.026774    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"
Nov 02 12:22:31.412520 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:22:31.412489    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4233ad_7e42_44ee_b826_6106d6278ee7.slice/crio-284e393dd03f1dbce28632b0942338659ff44e9600f13e6d2505b4ff20fbfe28 WatchSource:0}: Error finding container 284e393dd03f1dbce28632b0942338659ff44e9600f13e6d2505b4ff20fbfe28: Status 404 returned error can't find the container with id 284e393dd03f1dbce28632b0942338659ff44e9600f13e6d2505b4ff20fbfe28
Nov 02 12:22:31.414396 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:31.414367    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"]
Nov 02 12:22:31.591127 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:31.591100    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8897bb5d-ac09-4389-a3fe-65bae125f6be" path="/var/lib/kubelet/pods/8897bb5d-ac09-4389-a3fe-65bae125f6be/volumes"
Nov 02 12:22:31.591473 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:31.591461    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c2a8ed4-5986-40b4-a810-452e01712e38" path="/var/lib/kubelet/pods/9c2a8ed4-5986-40b4-a810-452e01712e38/volumes"
Nov 02 12:22:31.608496 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:31.608463    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm" event={"ID":"2b4233ad-7e42-44ee-b826-6106d6278ee7","Type":"ContainerStarted","Data":"284e393dd03f1dbce28632b0942338659ff44e9600f13e6d2505b4ff20fbfe28"}
Nov 02 12:22:31.728579 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:31.728555    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"
Nov 02 12:22:31.790291 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:31.790267    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cace0f3d-c048-4f89-9fb0-180afb8b468c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cace0f3d-c048-4f89-9fb0-180afb8b468c-maas-api-xks-ca\") pod \"cace0f3d-c048-4f89-9fb0-180afb8b468c\" (UID: \"cace0f3d-c048-4f89-9fb0-180afb8b468c\") "
Nov 02 12:22:31.790596 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:31.790344    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cace0f3d-c048-4f89-9fb0-180afb8b468c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cace0f3d-c048-4f89-9fb0-180afb8b468c-maas-api-openshift-ca\") pod \"cace0f3d-c048-4f89-9fb0-180afb8b468c\" (UID: \"cace0f3d-c048-4f89-9fb0-180afb8b468c\") "
Nov 02 12:22:31.790596 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:31.790480    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cace0f3d-c048-4f89-9fb0-180afb8b468c-kube-api-access-cljmn\" (UniqueName: \"kubernetes.io/projected/cace0f3d-c048-4f89-9fb0-180afb8b468c-kube-api-access-cljmn\") pod \"cace0f3d-c048-4f89-9fb0-180afb8b468c\" (UID: \"cace0f3d-c048-4f89-9fb0-180afb8b468c\") "
Nov 02 12:22:31.790695 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:31.790674    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cace0f3d-c048-4f89-9fb0-180afb8b468c-maas-api-openshift-ca" pod "cace0f3d-c048-4f89-9fb0-180afb8b468c" (UID: "cace0f3d-c048-4f89-9fb0-180afb8b468c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:22:31.790929 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:31.790904    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cace0f3d-c048-4f89-9fb0-180afb8b468c-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:31.792123 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:31.792101    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cace0f3d-c048-4f89-9fb0-180afb8b468c-maas-api-xks-ca" pod "cace0f3d-c048-4f89-9fb0-180afb8b468c" (UID: "cace0f3d-c048-4f89-9fb0-180afb8b468c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:22:31.792326 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:31.792309    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cace0f3d-c048-4f89-9fb0-180afb8b468c-kube-api-access-cljmn" pod "cace0f3d-c048-4f89-9fb0-180afb8b468c" (UID: "cace0f3d-c048-4f89-9fb0-180afb8b468c"). InnerVolumeSpecName "kube-api-access-cljmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:22:31.892425 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:31.892401    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cace0f3d-c048-4f89-9fb0-180afb8b468c-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:31.892425 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:31.892425    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cljmn\" (UniqueName: \"kubernetes.io/projected/cace0f3d-c048-4f89-9fb0-180afb8b468c-kube-api-access-cljmn\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:32.619137 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:32.619046    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"
Nov 02 12:22:32.619137 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:32.619083    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn" event={"ID":"cace0f3d-c048-4f89-9fb0-180afb8b468c","Type":"ContainerDied","Data":"8bdbcf77b09e417b1ef57b5d4d080e9121f79ec9e933e66d502c60f8ba5a4aa1"}
Nov 02 12:22:32.619137 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:32.619129    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bdbcf77b09e417b1ef57b5d4d080e9121f79ec9e933e66d502c60f8ba5a4aa1"
Nov 02 12:22:32.620827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:32.620803    4464 generic.go:356] "Generic (PLEG): container finished" podID="2b4233ad-7e42-44ee-b826-6106d6278ee7" containerID="f8984e9965b540c6c70f85b4aa70f89f10cd9c654215df9e0e7e6ae43b30c0cf" exitCode=0
Nov 02 12:22:32.620983 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:32.620869    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm" event={"ID":"2b4233ad-7e42-44ee-b826-6106d6278ee7","Type":"ContainerDied","Data":"f8984e9965b540c6c70f85b4aa70f89f10cd9c654215df9e0e7e6ae43b30c0cf"}
Nov 02 12:22:33.778844 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:33.778820    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"
Nov 02 12:22:33.915021 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:33.914933    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b4233ad-7e42-44ee-b826-6106d6278ee7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b4233ad-7e42-44ee-b826-6106d6278ee7-maas-api-xks-ca\") pod \"2b4233ad-7e42-44ee-b826-6106d6278ee7\" (UID: \"2b4233ad-7e42-44ee-b826-6106d6278ee7\") "
Nov 02 12:22:33.915172 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:33.915056    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b4233ad-7e42-44ee-b826-6106d6278ee7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b4233ad-7e42-44ee-b826-6106d6278ee7-maas-api-openshift-ca\") pod \"2b4233ad-7e42-44ee-b826-6106d6278ee7\" (UID: \"2b4233ad-7e42-44ee-b826-6106d6278ee7\") "
Nov 02 12:22:33.915172 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:33.915152    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b4233ad-7e42-44ee-b826-6106d6278ee7-kube-api-access-mdzcn\" (UniqueName: \"kubernetes.io/projected/2b4233ad-7e42-44ee-b826-6106d6278ee7-kube-api-access-mdzcn\") pod \"2b4233ad-7e42-44ee-b826-6106d6278ee7\" (UID: \"2b4233ad-7e42-44ee-b826-6106d6278ee7\") "
Nov 02 12:22:33.915422 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:33.915395    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b4233ad-7e42-44ee-b826-6106d6278ee7-maas-api-openshift-ca" pod "2b4233ad-7e42-44ee-b826-6106d6278ee7" (UID: "2b4233ad-7e42-44ee-b826-6106d6278ee7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:22:33.915676 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:33.915654    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b4233ad-7e42-44ee-b826-6106d6278ee7-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:33.917218 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:33.917195    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b4233ad-7e42-44ee-b826-6106d6278ee7-maas-api-xks-ca" pod "2b4233ad-7e42-44ee-b826-6106d6278ee7" (UID: "2b4233ad-7e42-44ee-b826-6106d6278ee7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:22:33.917324 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:33.917274    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4233ad-7e42-44ee-b826-6106d6278ee7-kube-api-access-mdzcn" pod "2b4233ad-7e42-44ee-b826-6106d6278ee7" (UID: "2b4233ad-7e42-44ee-b826-6106d6278ee7"). InnerVolumeSpecName "kube-api-access-mdzcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:22:34.016526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:34.016493    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b4233ad-7e42-44ee-b826-6106d6278ee7-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:34.016526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:34.016520    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdzcn\" (UniqueName: \"kubernetes.io/projected/2b4233ad-7e42-44ee-b826-6106d6278ee7-kube-api-access-mdzcn\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:34.636201 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:34.636169    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"
Nov 02 12:22:34.636394 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:34.636217    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm" event={"ID":"2b4233ad-7e42-44ee-b826-6106d6278ee7","Type":"ContainerDied","Data":"284e393dd03f1dbce28632b0942338659ff44e9600f13e6d2505b4ff20fbfe28"}
Nov 02 12:22:34.636394 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:34.636275    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="284e393dd03f1dbce28632b0942338659ff44e9600f13e6d2505b4ff20fbfe28"
Nov 02 12:22:37.101440 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:37.101407    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"]
Nov 02 12:22:37.151861 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:37.151834    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95eb25be-kmttt"]
Nov 02 12:22:37.585921 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:37.585877    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93497097-faf6-4ede-ab05-d5e1a398051a" path="/var/lib/kubelet/pods/93497097-faf6-4ede-ab05-d5e1a398051a/volumes"
Nov 02 12:22:39.041172 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.041136    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"]
Nov 02 12:22:39.041721 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.041708    4464 cpu_manager.go:414] "containerMap: removing container" podUID="cace0f3d-c048-4f89-9fb0-180afb8b468c" containerName="revoke-keys"
Nov 02 12:22:39.041767 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.041722    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cace0f3d-c048-4f89-9fb0-180afb8b468c" containerName="revoke-keys"
Nov 02 12:22:39.041767 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.041738    4464 cpu_manager.go:414] "containerMap: removing container" podUID="2b4233ad-7e42-44ee-b826-6106d6278ee7" containerName="revoke-keys"
Nov 02 12:22:39.041767 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.041743    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b4233ad-7e42-44ee-b826-6106d6278ee7" containerName="revoke-keys"
Nov 02 12:22:39.041862 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.041846    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="cace0f3d-c048-4f89-9fb0-180afb8b468c" containerName="revoke-keys"
Nov 02 12:22:39.041862 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.041856    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b4233ad-7e42-44ee-b826-6106d6278ee7" containerName="revoke-keys"
Nov 02 12:22:39.045351 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.045336    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"
Nov 02 12:22:39.056606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.056582    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"]
Nov 02 12:22:39.104998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.104224    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:22:39.104998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.104369    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:22:39.104998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.104458    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:22:39.108998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.105444    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-003bee73-dphsq"]
Nov 02 12:22:39.145333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.142200    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:22:39.149789 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.149758    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"]
Nov 02 12:22:39.236001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.233459    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a4c94e-3e0e-419d-95ed-01659117c396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6111b248-cpljf\" (UID: \"08a4c94e-3e0e-419d-95ed-01659117c396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"
Nov 02 12:22:39.236001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.233597    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a4c94e-3e0e-419d-95ed-01659117c396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6111b248-cpljf\" (UID: \"08a4c94e-3e0e-419d-95ed-01659117c396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"
Nov 02 12:22:39.236001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.233669    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwxnh\" (UniqueName: \"kubernetes.io/projected/08a4c94e-3e0e-419d-95ed-01659117c396-kube-api-access-nwxnh\") pod \"maas-api-revoke-sub-6111b248-cpljf\" (UID: \"08a4c94e-3e0e-419d-95ed-01659117c396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"
Nov 02 12:22:39.334591 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.334495    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a4c94e-3e0e-419d-95ed-01659117c396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6111b248-cpljf\" (UID: \"08a4c94e-3e0e-419d-95ed-01659117c396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"
Nov 02 12:22:39.334766 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.334647    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a4c94e-3e0e-419d-95ed-01659117c396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6111b248-cpljf\" (UID: \"08a4c94e-3e0e-419d-95ed-01659117c396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"
Nov 02 12:22:39.334766 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.334696    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwxnh\" (UniqueName: \"kubernetes.io/projected/08a4c94e-3e0e-419d-95ed-01659117c396-kube-api-access-nwxnh\") pod \"maas-api-revoke-sub-6111b248-cpljf\" (UID: \"08a4c94e-3e0e-419d-95ed-01659117c396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"
Nov 02 12:22:39.335335 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.335307    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a4c94e-3e0e-419d-95ed-01659117c396-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6111b248-cpljf\" (UID: \"08a4c94e-3e0e-419d-95ed-01659117c396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"
Nov 02 12:22:39.337006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.336970    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a4c94e-3e0e-419d-95ed-01659117c396-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6111b248-cpljf\" (UID: \"08a4c94e-3e0e-419d-95ed-01659117c396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"
Nov 02 12:22:39.397285 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.397262    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwxnh\" (UniqueName: \"kubernetes.io/projected/08a4c94e-3e0e-419d-95ed-01659117c396-kube-api-access-nwxnh\") pod \"maas-api-revoke-sub-6111b248-cpljf\" (UID: \"08a4c94e-3e0e-419d-95ed-01659117c396\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"
Nov 02 12:22:39.582471 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.582436    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c618798-05ff-4856-809a-377c963667ea" path="/var/lib/kubelet/pods/0c618798-05ff-4856-809a-377c963667ea/volumes"
Nov 02 12:22:39.668666 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:39.668576    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"
Nov 02 12:22:40.062306 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:22:40.062272    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a4c94e_3e0e_419d_95ed_01659117c396.slice/crio-5b8a0ffe92e8f17bb4ada9b21a1c380077568047b84cf65f02904c128f3b1b64 WatchSource:0}: Error finding container 5b8a0ffe92e8f17bb4ada9b21a1c380077568047b84cf65f02904c128f3b1b64: Status 404 returned error can't find the container with id 5b8a0ffe92e8f17bb4ada9b21a1c380077568047b84cf65f02904c128f3b1b64
Nov 02 12:22:40.062806 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:40.062561    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"]
Nov 02 12:22:40.715901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:40.715822    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf" event={"ID":"08a4c94e-3e0e-419d-95ed-01659117c396","Type":"ContainerStarted","Data":"7c5f350dfce1d97037251ad796e37a2aa886a46d5b37385716cf5d7937e7bad7"}
Nov 02 12:22:40.715901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:40.715885    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf" event={"ID":"08a4c94e-3e0e-419d-95ed-01659117c396","Type":"ContainerStarted","Data":"5b8a0ffe92e8f17bb4ada9b21a1c380077568047b84cf65f02904c128f3b1b64"}
Nov 02 12:22:40.832917 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:40.832853    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf" podStartSLOduration=2.832836585 podStartE2EDuration="2.832836585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:22:40.832581439 +0000 UTC m=+1589.396289842" watchObservedRunningTime="2026-11-02 12:22:40.832836585 +0000 UTC m=+1589.396544995"
Nov 02 12:22:41.723925 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:41.723890    4464 generic.go:356] "Generic (PLEG): container finished" podID="08a4c94e-3e0e-419d-95ed-01659117c396" containerID="7c5f350dfce1d97037251ad796e37a2aa886a46d5b37385716cf5d7937e7bad7" exitCode=0
Nov 02 12:22:41.724316 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:41.724003    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf" event={"ID":"08a4c94e-3e0e-419d-95ed-01659117c396","Type":"ContainerDied","Data":"7c5f350dfce1d97037251ad796e37a2aa886a46d5b37385716cf5d7937e7bad7"}
Nov 02 12:22:42.879858 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:42.879835    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"
Nov 02 12:22:42.978463 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:42.978432    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08a4c94e-3e0e-419d-95ed-01659117c396-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a4c94e-3e0e-419d-95ed-01659117c396-maas-api-openshift-ca\") pod \"08a4c94e-3e0e-419d-95ed-01659117c396\" (UID: \"08a4c94e-3e0e-419d-95ed-01659117c396\") "
Nov 02 12:22:42.978639 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:42.978496    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a4c94e-3e0e-419d-95ed-01659117c396-kube-api-access-nwxnh\" (UniqueName: \"kubernetes.io/projected/08a4c94e-3e0e-419d-95ed-01659117c396-kube-api-access-nwxnh\") pod \"08a4c94e-3e0e-419d-95ed-01659117c396\" (UID: \"08a4c94e-3e0e-419d-95ed-01659117c396\") "
Nov 02 12:22:42.978639 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:42.978518    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08a4c94e-3e0e-419d-95ed-01659117c396-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a4c94e-3e0e-419d-95ed-01659117c396-maas-api-xks-ca\") pod \"08a4c94e-3e0e-419d-95ed-01659117c396\" (UID: \"08a4c94e-3e0e-419d-95ed-01659117c396\") "
Nov 02 12:22:42.978897 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:42.978867    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a4c94e-3e0e-419d-95ed-01659117c396-maas-api-openshift-ca" pod "08a4c94e-3e0e-419d-95ed-01659117c396" (UID: "08a4c94e-3e0e-419d-95ed-01659117c396"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:22:42.980583 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:42.980560    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a4c94e-3e0e-419d-95ed-01659117c396-kube-api-access-nwxnh" pod "08a4c94e-3e0e-419d-95ed-01659117c396" (UID: "08a4c94e-3e0e-419d-95ed-01659117c396"). InnerVolumeSpecName "kube-api-access-nwxnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:22:42.980682 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:42.980660    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a4c94e-3e0e-419d-95ed-01659117c396-maas-api-xks-ca" pod "08a4c94e-3e0e-419d-95ed-01659117c396" (UID: "08a4c94e-3e0e-419d-95ed-01659117c396"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:22:43.084453 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:43.084407    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a4c94e-3e0e-419d-95ed-01659117c396-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:43.084453 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:43.084449    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwxnh\" (UniqueName: \"kubernetes.io/projected/08a4c94e-3e0e-419d-95ed-01659117c396-kube-api-access-nwxnh\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:43.084723 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:43.084468    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a4c94e-3e0e-419d-95ed-01659117c396-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:43.739971 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:43.739932    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"
Nov 02 12:22:43.739971 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:43.739958    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf" event={"ID":"08a4c94e-3e0e-419d-95ed-01659117c396","Type":"ContainerDied","Data":"5b8a0ffe92e8f17bb4ada9b21a1c380077568047b84cf65f02904c128f3b1b64"}
Nov 02 12:22:43.740247 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:43.740021    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b8a0ffe92e8f17bb4ada9b21a1c380077568047b84cf65f02904c128f3b1b64"
Nov 02 12:22:48.359934 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.359901    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-866d944846-nnbj2"]
Nov 02 12:22:48.360540 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.360524    4464 cpu_manager.go:414] "containerMap: removing container" podUID="08a4c94e-3e0e-419d-95ed-01659117c396" containerName="revoke-keys"
Nov 02 12:22:48.360586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.360541    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a4c94e-3e0e-419d-95ed-01659117c396" containerName="revoke-keys"
Nov 02 12:22:48.360655 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.360646    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a4c94e-3e0e-419d-95ed-01659117c396" containerName="revoke-keys"
Nov 02 12:22:48.372314 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.372278    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:22:48.428544 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.428504    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 12:22:48.428758 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.428555    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-15ea039108c9\""
Nov 02 12:22:48.479122 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.479052    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 12:22:48.479122 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.479054    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 12:22:48.479404 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.479051    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6278d2f81b78\""
Nov 02 12:22:48.479404 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.479215    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 12:22:48.479404 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.479277    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-fkwpc\""
Nov 02 12:22:48.483910 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.483886    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d944846-nnbj2"]
Nov 02 12:22:48.486798 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.486772    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-plugins-config-volume\") pod \"payload-processing-external-model-866d944846-nnbj2\" (UID: \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\") " pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:22:48.486935 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.486913    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials\") pod \"payload-processing-external-model-866d944846-nnbj2\" (UID: \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\") " pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:22:48.487031 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.486951    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75bw4\" (UniqueName: \"kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-kube-api-access-75bw4\") pod \"payload-processing-external-model-866d944846-nnbj2\" (UID: \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\") " pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:22:48.487121 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.487096    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-routing-overlay-volume\") pod \"payload-processing-external-model-866d944846-nnbj2\" (UID: \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\") " pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:22:48.560153 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.560123    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7224-62bbdb"]
Nov 02 12:22:48.564361 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.564344    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7224-62bbdb"
Nov 02 12:22:48.588286 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.588251    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials\") pod \"payload-processing-external-model-866d944846-nnbj2\" (UID: \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\") " pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:22:48.588438 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.588308    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75bw4\" (UniqueName: \"kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-kube-api-access-75bw4\") pod \"payload-processing-external-model-866d944846-nnbj2\" (UID: \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\") " pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:22:48.588438 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.588401    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gj4h\" (UniqueName: \"kubernetes.io/projected/cefffd53-b53f-4c10-83ed-fe55807c93d9-kube-api-access-6gj4h\") pod \"test-curl-7224-62bbdb\" (UID: \"cefffd53-b53f-4c10-83ed-fe55807c93d9\") " pod="opendatahub/test-curl-7224-62bbdb"
Nov 02 12:22:48.588524 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.588487    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-routing-overlay-volume\") pod \"payload-processing-external-model-866d944846-nnbj2\" (UID: \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\") " pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:22:48.588649 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.588628    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-plugins-config-volume\") pod \"payload-processing-external-model-866d944846-nnbj2\" (UID: \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\") " pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:22:48.589281 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.589255    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-routing-overlay-volume\") pod \"payload-processing-external-model-866d944846-nnbj2\" (UID: \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\") " pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:22:48.589325 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.589295    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-plugins-config-volume\") pod \"payload-processing-external-model-866d944846-nnbj2\" (UID: \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\") " pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:22:48.590701 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.590680    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials\") pod \"payload-processing-external-model-866d944846-nnbj2\" (UID: \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\") " pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:22:48.616763 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.616697    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7224-62bbdb"]
Nov 02 12:22:48.617715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.617690    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5lqpk\""
Nov 02 12:22:48.650791 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.650761    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75bw4\" (UniqueName: \"kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-kube-api-access-75bw4\") pod \"payload-processing-external-model-866d944846-nnbj2\" (UID: \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\") " pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:22:48.688566 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.688531    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:22:48.689953 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.689928    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gj4h\" (UniqueName: \"kubernetes.io/projected/cefffd53-b53f-4c10-83ed-fe55807c93d9-kube-api-access-6gj4h\") pod \"test-curl-7224-62bbdb\" (UID: \"cefffd53-b53f-4c10-83ed-fe55807c93d9\") " pod="opendatahub/test-curl-7224-62bbdb"
Nov 02 12:22:48.755191 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.755120    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gj4h\" (UniqueName: \"kubernetes.io/projected/cefffd53-b53f-4c10-83ed-fe55807c93d9-kube-api-access-6gj4h\") pod \"test-curl-7224-62bbdb\" (UID: \"cefffd53-b53f-4c10-83ed-fe55807c93d9\") " pod="opendatahub/test-curl-7224-62bbdb"
Nov 02 12:22:48.796311 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.796276    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"]
Nov 02 12:22:48.803198 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.803165    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fbf1444a-nzdx5"]
Nov 02 12:22:48.856948 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.856922    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"]
Nov 02 12:22:48.857467 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.857437    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt" podUID="61efc388-83a8-4585-8399-815dea8f96bf" containerName="istio-proxy" containerID="cri-o://6b89c66cecd382878d6cf2d193d6e4f8eb876399f7e471171827e73110b2a013" gracePeriod=30
Nov 02 12:22:48.877454 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:48.877369    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7224-62bbdb"
Nov 02 12:22:49.157086 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:49.157058    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d944846-nnbj2"]
Nov 02 12:22:49.263314 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:22:49.263273    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcefffd53_b53f_4c10_83ed_fe55807c93d9.slice/crio-f05a850f9ec52d8bbd3ff9bb3a3b6a0afb71b09b1f10abd9168e929150c4e96a WatchSource:0}: Error finding container f05a850f9ec52d8bbd3ff9bb3a3b6a0afb71b09b1f10abd9168e929150c4e96a: Status 404 returned error can't find the container with id f05a850f9ec52d8bbd3ff9bb3a3b6a0afb71b09b1f10abd9168e929150c4e96a
Nov 02 12:22:49.263576 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:49.263553    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7224-62bbdb"]
Nov 02 12:22:49.265198 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:49.265178    4464 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:22:49.586254 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:49.586217    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363e0725-226b-4e32-996c-7023534370dd" path="/var/lib/kubelet/pods/363e0725-226b-4e32-996c-7023534370dd/volumes"
Nov 02 12:22:49.799577 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:49.799456    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2" event={"ID":"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac","Type":"ContainerStarted","Data":"1c843bcd4cc5a173c66216d8892e3db90897feb8d66956a13cba2be0ec2a01c2"}
Nov 02 12:22:49.799577 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:49.799521    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2" event={"ID":"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac","Type":"ContainerStarted","Data":"91ceeae6d91d500e7337d440f84d31ff4255b7ebcae27b6c65613db74cc729a9"}
Nov 02 12:22:49.799875 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:49.799858    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:22:49.801681 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:49.801653    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7224-62bbdb" event={"ID":"cefffd53-b53f-4c10-83ed-fe55807c93d9","Type":"ContainerStarted","Data":"f05a850f9ec52d8bbd3ff9bb3a3b6a0afb71b09b1f10abd9168e929150c4e96a"}
Nov 02 12:22:49.936921 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:49.936814    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2" podStartSLOduration=1.936797716 podStartE2EDuration="1.936797716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:22:49.931705722 +0000 UTC m=+1598.495414155" watchObservedRunningTime="2026-11-02 12:22:49.936797716 +0000 UTC m=+1598.500506124"
Nov 02 12:22:50.811010 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:50.810954    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7224-62bbdb" event={"ID":"cefffd53-b53f-4c10-83ed-fe55807c93d9","Type":"ContainerStarted","Data":"5d999985e733a535ec6fdb6ae6f3b577e9d8e4297762489d7e316fea76049b1a"}
Nov 02 12:22:52.461042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:52.460967    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7224-62bbdb" podStartSLOduration=3.384591061 podStartE2EDuration="4.460954155s" totalImagesPullingTime="1.076363094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:22:50.929159699 +0000 UTC m=+1599.492868110" watchObservedRunningTime="2026-11-02 12:22:52.460954155 +0000 UTC m=+1601.024662563"
Nov 02 12:22:52.464382 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:52.464352    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7224-62bbdb"]
Nov 02 12:22:52.468824 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:52.468798    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7224-62bbdb"]
Nov 02 12:22:52.727506 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:52.727472    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:22:52.837137 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:52.837084    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7224-62bbdb" podUID="cefffd53-b53f-4c10-83ed-fe55807c93d9" containerName="test-curl-7224-62bbdb" containerID="cri-o://5d999985e733a535ec6fdb6ae6f3b577e9d8e4297762489d7e316fea76049b1a" gracePeriod=2
Nov 02 12:22:52.889316 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:22:52.889281    4464 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7224-62bbdb\" is forbidden: User \"system:node:ip-10-0-133-239.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-239.us-west-1.compute.internal' and this object" podUID="cefffd53-b53f-4c10-83ed-fe55807c93d9" pod="opendatahub/test-curl-7224-62bbdb"
Nov 02 12:22:53.486000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.483785    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"]
Nov 02 12:22:53.486000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.484822    4464 cpu_manager.go:414] "containerMap: removing container" podUID="cefffd53-b53f-4c10-83ed-fe55807c93d9" containerName="test-curl-7224-62bbdb"
Nov 02 12:22:53.486000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.484844    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cefffd53-b53f-4c10-83ed-fe55807c93d9" containerName="test-curl-7224-62bbdb"
Nov 02 12:22:53.486000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.485027    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="cefffd53-b53f-4c10-83ed-fe55807c93d9" containerName="test-curl-7224-62bbdb"
Nov 02 12:22:53.490656 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.490637    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"
Nov 02 12:22:53.529407 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.529373    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"]
Nov 02 12:22:53.628392 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.628353    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:22:53.628782 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.628760    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:22:53.629539 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.629519    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:22:53.629739 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.629721    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:22:53.659031 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.650338    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a432197-g8dw2\" (UID: \"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"
Nov 02 12:22:53.659031 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.650388    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a432197-g8dw2\" (UID: \"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"
Nov 02 12:22:53.659031 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.650433    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmzc2\" (UniqueName: \"kubernetes.io/projected/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-kube-api-access-kmzc2\") pod \"maas-api-revoke-keys-e2e-mt-6a432197-g8dw2\" (UID: \"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"
Nov 02 12:22:53.749366 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:22:53.749279    4464 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7224-62bbdb\" is forbidden: User \"system:node:ip-10-0-133-239.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-239.us-west-1.compute.internal' and this object" podUID="cefffd53-b53f-4c10-83ed-fe55807c93d9" pod="opendatahub/test-curl-7224-62bbdb"
Nov 02 12:22:53.818035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.809343    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a432197-g8dw2\" (UID: \"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"
Nov 02 12:22:53.818035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.809463    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmzc2\" (UniqueName: \"kubernetes.io/projected/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-kube-api-access-kmzc2\") pod \"maas-api-revoke-keys-e2e-mt-6a432197-g8dw2\" (UID: \"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"
Nov 02 12:22:53.818035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.809690    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a432197-g8dw2\" (UID: \"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"
Nov 02 12:22:53.823527 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.823488    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a432197-g8dw2\" (UID: \"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"
Nov 02 12:22:53.880999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.880479    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6a432197-g8dw2\" (UID: \"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"
Nov 02 12:22:53.909888 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:53.909863    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmzc2\" (UniqueName: \"kubernetes.io/projected/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-kube-api-access-kmzc2\") pod \"maas-api-revoke-keys-e2e-mt-6a432197-g8dw2\" (UID: \"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"
Nov 02 12:22:54.054578 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:54.054509    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"]
Nov 02 12:22:54.060199 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:54.060163    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-306303de-cvn92"]
Nov 02 12:22:54.117480 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:54.117194    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"
Nov 02 12:22:55.027485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.026942    4464 generic.go:356] "Generic (PLEG): container finished" podID="61efc388-83a8-4585-8399-815dea8f96bf" containerID="6b89c66cecd382878d6cf2d193d6e4f8eb876399f7e471171827e73110b2a013" exitCode=0
Nov 02 12:22:55.027485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.027295    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt" event={"ID":"61efc388-83a8-4585-8399-815dea8f96bf","Type":"ContainerDied","Data":"6b89c66cecd382878d6cf2d193d6e4f8eb876399f7e471171827e73110b2a013"}
Nov 02 12:22:55.027485 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.027360    4464 scope.go:122] "RemoveContainer" containerID="ffeeca88a314e080740ea9fc8affcfa67df5a511afd3341e9ab03b38ef1662f8"
Nov 02 12:22:55.121498 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.121301    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:22:55.258726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.258686    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-workload-certs\") pod \"61efc388-83a8-4585-8399-815dea8f96bf\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") "
Nov 02 12:22:55.258915 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.258763    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-credential-socket\") pod \"61efc388-83a8-4585-8399-815dea8f96bf\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") "
Nov 02 12:22:55.258915 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.258850    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61efc388-83a8-4585-8399-815dea8f96bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61efc388-83a8-4585-8399-815dea8f96bf-istio-podinfo\") pod \"61efc388-83a8-4585-8399-815dea8f96bf\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") "
Nov 02 12:22:55.259078 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.258990    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61efc388-83a8-4585-8399-815dea8f96bf-kube-api-access-n265m\" (UniqueName: \"kubernetes.io/projected/61efc388-83a8-4585-8399-815dea8f96bf-kube-api-access-n265m\") pod \"61efc388-83a8-4585-8399-815dea8f96bf\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") "
Nov 02 12:22:55.259078 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.259024    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61efc388-83a8-4585-8399-815dea8f96bf-istio-token\" (UniqueName: \"kubernetes.io/projected/61efc388-83a8-4585-8399-815dea8f96bf-istio-token\") pod \"61efc388-83a8-4585-8399-815dea8f96bf\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") "
Nov 02 12:22:55.259078 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.259038    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-workload-certs" pod "61efc388-83a8-4585-8399-815dea8f96bf" (UID: "61efc388-83a8-4585-8399-815dea8f96bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:22:55.259078 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.259063    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-istio-envoy\") pod \"61efc388-83a8-4585-8399-815dea8f96bf\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") "
Nov 02 12:22:55.259291 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.259148    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-workload-socket\") pod \"61efc388-83a8-4585-8399-815dea8f96bf\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") "
Nov 02 12:22:55.259291 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.259229    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-istio-data\") pod \"61efc388-83a8-4585-8399-815dea8f96bf\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") "
Nov 02 12:22:55.259291 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.259276    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61efc388-83a8-4585-8399-815dea8f96bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61efc388-83a8-4585-8399-815dea8f96bf-istiod-ca-cert\") pod \"61efc388-83a8-4585-8399-815dea8f96bf\" (UID: \"61efc388-83a8-4585-8399-815dea8f96bf\") "
Nov 02 12:22:55.259444 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.259418    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-credential-socket" pod "61efc388-83a8-4585-8399-815dea8f96bf" (UID: "61efc388-83a8-4585-8399-815dea8f96bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:22:55.259803 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.259778    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61efc388-83a8-4585-8399-815dea8f96bf-istiod-ca-cert" pod "61efc388-83a8-4585-8399-815dea8f96bf" (UID: "61efc388-83a8-4585-8399-815dea8f96bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:22:55.259910 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.259893    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:55.260021 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.259998    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-workload-socket" pod "61efc388-83a8-4585-8399-815dea8f96bf" (UID: "61efc388-83a8-4585-8399-815dea8f96bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:22:55.260097 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.259918    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61efc388-83a8-4585-8399-815dea8f96bf-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:55.260097 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.260043    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:55.260281 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.260253    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-istio-data" pod "61efc388-83a8-4585-8399-815dea8f96bf" (UID: "61efc388-83a8-4585-8399-815dea8f96bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:22:55.261607 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.261580    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-istio-envoy" pod "61efc388-83a8-4585-8399-815dea8f96bf" (UID: "61efc388-83a8-4585-8399-815dea8f96bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:22:55.268343 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.268318    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61efc388-83a8-4585-8399-815dea8f96bf-istio-podinfo" pod "61efc388-83a8-4585-8399-815dea8f96bf" (UID: "61efc388-83a8-4585-8399-815dea8f96bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:22:55.268513 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.268491    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61efc388-83a8-4585-8399-815dea8f96bf-kube-api-access-n265m" pod "61efc388-83a8-4585-8399-815dea8f96bf" (UID: "61efc388-83a8-4585-8399-815dea8f96bf"). InnerVolumeSpecName "kube-api-access-n265m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:22:55.270259 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.270236    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61efc388-83a8-4585-8399-815dea8f96bf-istio-token" pod "61efc388-83a8-4585-8399-815dea8f96bf" (UID: "61efc388-83a8-4585-8399-815dea8f96bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:22:55.348442 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:22:55.348409    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ecc964_fea9_4220_bdf0_c2e5d4fa6cc3.slice/crio-448b11037f557ee42c9f254a08e3a4e52be7768f160cac61f6bd4b0bda81044d WatchSource:0}: Error finding container 448b11037f557ee42c9f254a08e3a4e52be7768f160cac61f6bd4b0bda81044d: Status 404 returned error can't find the container with id 448b11037f557ee42c9f254a08e3a4e52be7768f160cac61f6bd4b0bda81044d
Nov 02 12:22:55.352612 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.352587    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"]
Nov 02 12:22:55.361638 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.361615    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:55.361638 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.361639    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:55.361761 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.361649    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61efc388-83a8-4585-8399-815dea8f96bf-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:55.361761 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.361658    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n265m\" (UniqueName: \"kubernetes.io/projected/61efc388-83a8-4585-8399-815dea8f96bf-kube-api-access-n265m\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:55.361761 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.361668    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61efc388-83a8-4585-8399-815dea8f96bf-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:55.361761 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.361676    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61efc388-83a8-4585-8399-815dea8f96bf-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:55.363583 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.363563    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7224-62bbdb"
Nov 02 12:22:55.563049 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.563013    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cefffd53-b53f-4c10-83ed-fe55807c93d9-kube-api-access-6gj4h\" (UniqueName: \"kubernetes.io/projected/cefffd53-b53f-4c10-83ed-fe55807c93d9-kube-api-access-6gj4h\") pod \"cefffd53-b53f-4c10-83ed-fe55807c93d9\" (UID: \"cefffd53-b53f-4c10-83ed-fe55807c93d9\") "
Nov 02 12:22:55.565188 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.565163    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefffd53-b53f-4c10-83ed-fe55807c93d9-kube-api-access-6gj4h" pod "cefffd53-b53f-4c10-83ed-fe55807c93d9" (UID: "cefffd53-b53f-4c10-83ed-fe55807c93d9"). InnerVolumeSpecName "kube-api-access-6gj4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:22:55.582933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.582901    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b67762-1dbe-464d-b137-d0392e00781d" path="/var/lib/kubelet/pods/65b67762-1dbe-464d-b137-d0392e00781d/volumes"
Nov 02 12:22:55.583520 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.583497    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cefffd53-b53f-4c10-83ed-fe55807c93d9" path="/var/lib/kubelet/pods/cefffd53-b53f-4c10-83ed-fe55807c93d9/volumes"
Nov 02 12:22:55.664677 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:55.664598    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gj4h\" (UniqueName: \"kubernetes.io/projected/cefffd53-b53f-4c10-83ed-fe55807c93d9-kube-api-access-6gj4h\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:56.036908 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:56.036865    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2" event={"ID":"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3","Type":"ContainerStarted","Data":"6cc2c2f06ee9ea6d364bb2cb7de3a4c5f0d3c25b228c386382b96837cce600af"}
Nov 02 12:22:56.037453 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:56.036930    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2" event={"ID":"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3","Type":"ContainerStarted","Data":"448b11037f557ee42c9f254a08e3a4e52be7768f160cac61f6bd4b0bda81044d"}
Nov 02 12:22:56.038182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:56.038158    4464 generic.go:356] "Generic (PLEG): container finished" podID="cefffd53-b53f-4c10-83ed-fe55807c93d9" containerID="5d999985e733a535ec6fdb6ae6f3b577e9d8e4297762489d7e316fea76049b1a" exitCode=137
Nov 02 12:22:56.038246 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:56.038194    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7224-62bbdb"
Nov 02 12:22:56.038292 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:56.038254    4464 scope.go:122] "RemoveContainer" containerID="5d999985e733a535ec6fdb6ae6f3b577e9d8e4297762489d7e316fea76049b1a"
Nov 02 12:22:56.039955 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:56.039931    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt" event={"ID":"61efc388-83a8-4585-8399-815dea8f96bf","Type":"ContainerDied","Data":"1a67052a0a6c3060a63f91287e7575ef4a43fe6e3b33dac648ade538e82336dd"}
Nov 02 12:22:56.040132 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:56.039956    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"
Nov 02 12:22:56.048671 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:56.048655    4464 scope.go:122] "RemoveContainer" containerID="5d999985e733a535ec6fdb6ae6f3b577e9d8e4297762489d7e316fea76049b1a"
Nov 02 12:22:56.048954 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:22:56.048934    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d999985e733a535ec6fdb6ae6f3b577e9d8e4297762489d7e316fea76049b1a\": container with ID starting with 5d999985e733a535ec6fdb6ae6f3b577e9d8e4297762489d7e316fea76049b1a not found: ID does not exist" containerID="5d999985e733a535ec6fdb6ae6f3b577e9d8e4297762489d7e316fea76049b1a"
Nov 02 12:22:56.049054 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:56.048960    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d999985e733a535ec6fdb6ae6f3b577e9d8e4297762489d7e316fea76049b1a"} err="failed to get container status \"5d999985e733a535ec6fdb6ae6f3b577e9d8e4297762489d7e316fea76049b1a\": rpc error: code = NotFound desc = could not find container \"5d999985e733a535ec6fdb6ae6f3b577e9d8e4297762489d7e316fea76049b1a\": container with ID starting with 5d999985e733a535ec6fdb6ae6f3b577e9d8e4297762489d7e316fea76049b1a not found: ID does not exist"
Nov 02 12:22:56.049054 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:56.048993    4464 scope.go:122] "RemoveContainer" containerID="6b89c66cecd382878d6cf2d193d6e4f8eb876399f7e471171827e73110b2a013"
Nov 02 12:22:56.155886 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:56.155832    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2" podStartSLOduration=3.155817414 podStartE2EDuration="3.155817414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:22:56.154064912 +0000 UTC m=+1604.717773322" watchObservedRunningTime="2026-11-02 12:22:56.155817414 +0000 UTC m=+1604.719525822"
Nov 02 12:22:56.311830 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:56.311741    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"]
Nov 02 12:22:56.359317 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:56.359219    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fbf1444a-openshift-default-78d99bcf77-x2vzt"]
Nov 02 12:22:57.053456 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:57.053423    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"]
Nov 02 12:22:57.056397 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:57.056367    4464 generic.go:356] "Generic (PLEG): container finished" podID="24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3" containerID="6cc2c2f06ee9ea6d364bb2cb7de3a4c5f0d3c25b228c386382b96837cce600af" exitCode=0
Nov 02 12:22:57.056557 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:57.056457    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2" event={"ID":"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3","Type":"ContainerDied","Data":"6cc2c2f06ee9ea6d364bb2cb7de3a4c5f0d3c25b228c386382b96837cce600af"}
Nov 02 12:22:57.057675 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:57.057655    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba1c4212-qrcpr"]
Nov 02 12:22:57.583102 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:57.583073    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61efc388-83a8-4585-8399-815dea8f96bf" path="/var/lib/kubelet/pods/61efc388-83a8-4585-8399-815dea8f96bf/volumes"
Nov 02 12:22:57.583512 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:57.583499    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc135360-afd8-4b66-97fd-92d9f60c8611" path="/var/lib/kubelet/pods/fc135360-afd8-4b66-97fd-92d9f60c8611/volumes"
Nov 02 12:22:57.998553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:57.998513    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"]
Nov 02 12:22:57.999184 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:57.999168    4464 cpu_manager.go:414] "containerMap: removing container" podUID="61efc388-83a8-4585-8399-815dea8f96bf" containerName="istio-proxy"
Nov 02 12:22:57.999184 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:57.999185    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61efc388-83a8-4585-8399-815dea8f96bf" containerName="istio-proxy"
Nov 02 12:22:57.999328 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:57.999319    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="61efc388-83a8-4585-8399-815dea8f96bf" containerName="istio-proxy"
Nov 02 12:22:58.002818 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.002802    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"
Nov 02 12:22:58.060661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.060629    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"]
Nov 02 12:22:58.094222 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.094187    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7814e6-b986-4c57-8c69-91310ce6e283-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq\" (UID: \"4e7814e6-b986-4c57-8c69-91310ce6e283\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"
Nov 02 12:22:58.094412 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.094254    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc7w8\" (UniqueName: \"kubernetes.io/projected/4e7814e6-b986-4c57-8c69-91310ce6e283-kube-api-access-xc7w8\") pod \"payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq\" (UID: \"4e7814e6-b986-4c57-8c69-91310ce6e283\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"
Nov 02 12:22:58.131999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.111054    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-1cc3a7b3-dockercfg-crctf\""
Nov 02 12:22:58.131999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.111176    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-1cc3a7b3\""
Nov 02 12:22:58.224606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.198006    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7814e6-b986-4c57-8c69-91310ce6e283-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq\" (UID: \"4e7814e6-b986-4c57-8c69-91310ce6e283\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"
Nov 02 12:22:58.224606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.198137    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc7w8\" (UniqueName: \"kubernetes.io/projected/4e7814e6-b986-4c57-8c69-91310ce6e283-kube-api-access-xc7w8\") pod \"payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq\" (UID: \"4e7814e6-b986-4c57-8c69-91310ce6e283\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"
Nov 02 12:22:58.224606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.199705    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7814e6-b986-4c57-8c69-91310ce6e283-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq\" (UID: \"4e7814e6-b986-4c57-8c69-91310ce6e283\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"
Nov 02 12:22:58.224606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.203965    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"]
Nov 02 12:22:58.224606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.205241    4464 cpu_manager.go:414] "containerMap: removing container" podUID="61efc388-83a8-4585-8399-815dea8f96bf" containerName="istio-proxy"
Nov 02 12:22:58.224606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.205260    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61efc388-83a8-4585-8399-815dea8f96bf" containerName="istio-proxy"
Nov 02 12:22:58.224606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.205423    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="61efc388-83a8-4585-8399-815dea8f96bf" containerName="istio-proxy"
Nov 02 12:22:58.224606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.213678    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"
Nov 02 12:22:58.225816 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.225767    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"]
Nov 02 12:22:58.260786 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.260757    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc7w8\" (UniqueName: \"kubernetes.io/projected/4e7814e6-b986-4c57-8c69-91310ce6e283-kube-api-access-xc7w8\") pod \"payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq\" (UID: \"4e7814e6-b986-4c57-8c69-91310ce6e283\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"
Nov 02 12:22:58.296014 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:22:58.295789    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:22:58.299664 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.299633    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc9abd9-14d6-462a-90f8-a486d9270087-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g\" (UID: \"bfc9abd9-14d6-462a-90f8-a486d9270087\") " pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"
Nov 02 12:22:58.299915 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.299880    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm954\" (UniqueName: \"kubernetes.io/projected/bfc9abd9-14d6-462a-90f8-a486d9270087-kube-api-access-nm954\") pod \"payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g\" (UID: \"bfc9abd9-14d6-462a-90f8-a486d9270087\") " pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"
Nov 02 12:22:58.317003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.316967    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"
Nov 02 12:22:58.361195 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.361160    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"
Nov 02 12:22:58.400773 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.400745    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-maas-api-xks-ca\") pod \"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3\" (UID: \"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3\") "
Nov 02 12:22:58.400903 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.400850    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-kube-api-access-kmzc2\" (UniqueName: \"kubernetes.io/projected/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-kube-api-access-kmzc2\") pod \"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3\" (UID: \"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3\") "
Nov 02 12:22:58.401117 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.401096    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm954\" (UniqueName: \"kubernetes.io/projected/bfc9abd9-14d6-462a-90f8-a486d9270087-kube-api-access-nm954\") pod \"payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g\" (UID: \"bfc9abd9-14d6-462a-90f8-a486d9270087\") " pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"
Nov 02 12:22:58.402329 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.401266    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc9abd9-14d6-462a-90f8-a486d9270087-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g\" (UID: \"bfc9abd9-14d6-462a-90f8-a486d9270087\") " pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"
Nov 02 12:22:58.402329 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.402282    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc9abd9-14d6-462a-90f8-a486d9270087-plugins-config-volume\") pod \"payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g\" (UID: \"bfc9abd9-14d6-462a-90f8-a486d9270087\") " pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"
Nov 02 12:22:58.403816 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.403793    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-maas-api-xks-ca" pod "24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3" (UID: "24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:22:58.404883 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.404857    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-kube-api-access-kmzc2" pod "24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3" (UID: "24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3"). InnerVolumeSpecName "kube-api-access-kmzc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:22:58.463598 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.463564    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm954\" (UniqueName: \"kubernetes.io/projected/bfc9abd9-14d6-462a-90f8-a486d9270087-kube-api-access-nm954\") pod \"payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g\" (UID: \"bfc9abd9-14d6-462a-90f8-a486d9270087\") " pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"
Nov 02 12:22:58.501909 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.501879    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-maas-api-openshift-ca\") pod \"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3\" (UID: \"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3\") "
Nov 02 12:22:58.502241 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.502226    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:58.502241 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.502242    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmzc2\" (UniqueName: \"kubernetes.io/projected/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-kube-api-access-kmzc2\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:58.502345 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.502238    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-maas-api-openshift-ca" pod "24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3" (UID: "24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:22:58.544660 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.544599    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"
Nov 02 12:22:58.605172 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.603490    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:22:58.700167 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:22:58.700132    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7814e6_b986_4c57_8c69_91310ce6e283.slice/crio-1b2a6d447d44616367a65e4addd288626379cbcb77dc51992462dcfffc911f67 WatchSource:0}: Error finding container 1b2a6d447d44616367a65e4addd288626379cbcb77dc51992462dcfffc911f67: Status 404 returned error can't find the container with id 1b2a6d447d44616367a65e4addd288626379cbcb77dc51992462dcfffc911f67
Nov 02 12:22:58.701013 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.700989    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"]
Nov 02 12:22:58.922682 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:22:58.922646    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc9abd9_14d6_462a_90f8_a486d9270087.slice/crio-d2a2d48af95b801f51a422cafd747a2a2e3f2672f4a5b8c803b51281a62b2cea WatchSource:0}: Error finding container d2a2d48af95b801f51a422cafd747a2a2e3f2672f4a5b8c803b51281a62b2cea: Status 404 returned error can't find the container with id d2a2d48af95b801f51a422cafd747a2a2e3f2672f4a5b8c803b51281a62b2cea
Nov 02 12:22:58.922968 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:58.922941    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"]
Nov 02 12:22:59.093644 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:59.093602    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq" event={"ID":"4e7814e6-b986-4c57-8c69-91310ce6e283","Type":"ContainerStarted","Data":"9c3c46e8080118aff1e36c5af62128e9386474f6a40440bd3f7c382e9739f72f"}
Nov 02 12:22:59.094212 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:59.094188    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"
Nov 02 12:22:59.094338 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:59.094224    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq" event={"ID":"4e7814e6-b986-4c57-8c69-91310ce6e283","Type":"ContainerStarted","Data":"1b2a6d447d44616367a65e4addd288626379cbcb77dc51992462dcfffc911f67"}
Nov 02 12:22:59.095322 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:59.095296    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g" event={"ID":"bfc9abd9-14d6-462a-90f8-a486d9270087","Type":"ContainerStarted","Data":"646fbbff6487e4624066cee9a8f0ca2447d1fa01026c7bfbc7ed4edb859824ba"}
Nov 02 12:22:59.095418 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:59.095351    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g" event={"ID":"bfc9abd9-14d6-462a-90f8-a486d9270087","Type":"ContainerStarted","Data":"d2a2d48af95b801f51a422cafd747a2a2e3f2672f4a5b8c803b51281a62b2cea"}
Nov 02 12:22:59.095418 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:59.095381    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"
Nov 02 12:22:59.096761 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:59.096745    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"
Nov 02 12:22:59.096852 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:59.096787    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2" event={"ID":"24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3","Type":"ContainerDied","Data":"448b11037f557ee42c9f254a08e3a4e52be7768f160cac61f6bd4b0bda81044d"}
Nov 02 12:22:59.096852 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:59.096840    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="448b11037f557ee42c9f254a08e3a4e52be7768f160cac61f6bd4b0bda81044d"
Nov 02 12:22:59.212787 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:59.212683    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq" podStartSLOduration=2.212669109 podStartE2EDuration="2.212669109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:22:59.211689966 +0000 UTC m=+1607.775398386" watchObservedRunningTime="2026-11-02 12:22:59.212669109 +0000 UTC m=+1607.776377517"
Nov 02 12:22:59.523997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:22:59.518312    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g" podStartSLOduration=1.518290451 podStartE2EDuration="1.518290451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:22:59.388412858 +0000 UTC m=+1607.952121270" watchObservedRunningTime="2026-11-02 12:22:59.518290451 +0000 UTC m=+1608.081998860"
Nov 02 12:23:00.814111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:00.814082    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:23:04.974472 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:04.974435    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"]
Nov 02 12:23:05.087104 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:05.087055    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"]
Nov 02 12:23:05.090999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:05.090941    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6672e2a0-nwwjx"]
Nov 02 12:23:05.095172 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:05.095146    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"]
Nov 02 12:23:05.583144 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:05.583113    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7db0ecee-4d9e-4639-9967-bb61236d29a4" path="/var/lib/kubelet/pods/7db0ecee-4d9e-4639-9967-bb61236d29a4/volumes"
Nov 02 12:23:09.975139 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:09.975095    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4" podUID="9c76562c-76e1-4367-a648-59f93a59e7ba" containerName="payload-processing" containerID="cri-o://96812a29a69c3e2bbf0e80ad7ce3056647a3cf88c3c73ec476ad25be501f5761" gracePeriod=25
Nov 02 12:23:10.096365 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:10.096324    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g" podUID="f1252145-23ce-4c2c-bc8f-26fd85eb14f9" containerName="payload-pre-processing" containerID="cri-o://3b14d8086fa48f12f43942c6ae66f8a36e74d7d11fe0c7aedde9014ad32ade0d" gracePeriod=25
Nov 02 12:23:10.107300 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:10.107276    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"
Nov 02 12:23:10.107425 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:10.107340    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"
Nov 02 12:23:11.933330 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:11.933298    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:23:12.527424 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:12.527388    4464 scope.go:122] "RemoveContainer" containerID="5bff59ff8da5bd7be4ce8d1364bfa0882692fdfdd62a5d081a114edac3bc2989"
Nov 02 12:23:12.540071 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:12.540042    4464 scope.go:122] "RemoveContainer" containerID="9129fa7db4e9d9af2ac358bd56973cf84547583b2b5d02789ac136d6a76aadc5"
Nov 02 12:23:12.550551 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:12.550526    4464 scope.go:122] "RemoveContainer" containerID="765c7696a84f757f0ce867bb1bcb2704ea5db0bb15227d6030a5b70926b44e44"
Nov 02 12:23:12.560106 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:12.560080    4464 scope.go:122] "RemoveContainer" containerID="b68922ac35d776cc3e9105e01f84455d93acef8c4341cf121639f2d97b2ae861"
Nov 02 12:23:17.630138 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:17.630102    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4" podUID="9c76562c-76e1-4367-a648-59f93a59e7ba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:23:17.630138 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:17.630120    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g" podUID="f1252145-23ce-4c2c-bc8f-26fd85eb14f9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:23:18.792371 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.792335    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"]
Nov 02 12:23:18.793243 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.793221    4464 cpu_manager.go:414] "containerMap: removing container" podUID="24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3" containerName="revoke-keys"
Nov 02 12:23:18.793243 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.793242    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3" containerName="revoke-keys"
Nov 02 12:23:18.793429 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.793398    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3" containerName="revoke-keys"
Nov 02 12:23:18.798843 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.798822    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"
Nov 02 12:23:18.852714 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.852685    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:23:18.854193 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.852725    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:23:18.854193 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.852788    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:23:18.854193 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.852826    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:23:18.855265 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.855244    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"]
Nov 02 12:23:18.893403 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.893371    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dbd750-b043-4950-b08c-ef027ef51ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-962c869b-hx6f7\" (UID: \"e9dbd750-b043-4950-b08c-ef027ef51ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"
Nov 02 12:23:18.893403 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.893415    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dbd750-b043-4950-b08c-ef027ef51ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-962c869b-hx6f7\" (UID: \"e9dbd750-b043-4950-b08c-ef027ef51ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"
Nov 02 12:23:18.893616 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.893493    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfkz9\" (UniqueName: \"kubernetes.io/projected/e9dbd750-b043-4950-b08c-ef027ef51ef8-kube-api-access-lfkz9\") pod \"maas-api-revoke-sub-962c869b-hx6f7\" (UID: \"e9dbd750-b043-4950-b08c-ef027ef51ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"
Nov 02 12:23:18.994924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.994886    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dbd750-b043-4950-b08c-ef027ef51ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-962c869b-hx6f7\" (UID: \"e9dbd750-b043-4950-b08c-ef027ef51ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"
Nov 02 12:23:18.994924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.994927    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dbd750-b043-4950-b08c-ef027ef51ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-962c869b-hx6f7\" (UID: \"e9dbd750-b043-4950-b08c-ef027ef51ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"
Nov 02 12:23:18.995152 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.994945    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfkz9\" (UniqueName: \"kubernetes.io/projected/e9dbd750-b043-4950-b08c-ef027ef51ef8-kube-api-access-lfkz9\") pod \"maas-api-revoke-sub-962c869b-hx6f7\" (UID: \"e9dbd750-b043-4950-b08c-ef027ef51ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"
Nov 02 12:23:18.995677 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.995650    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dbd750-b043-4950-b08c-ef027ef51ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-962c869b-hx6f7\" (UID: \"e9dbd750-b043-4950-b08c-ef027ef51ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"
Nov 02 12:23:18.997260 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:18.997243    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dbd750-b043-4950-b08c-ef027ef51ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-962c869b-hx6f7\" (UID: \"e9dbd750-b043-4950-b08c-ef027ef51ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"
Nov 02 12:23:19.056909 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:19.056845    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfkz9\" (UniqueName: \"kubernetes.io/projected/e9dbd750-b043-4950-b08c-ef027ef51ef8-kube-api-access-lfkz9\") pod \"maas-api-revoke-sub-962c869b-hx6f7\" (UID: \"e9dbd750-b043-4950-b08c-ef027ef51ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"
Nov 02 12:23:19.112816 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:19.112788    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"
Nov 02 12:23:19.492683 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:23:19.492651    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9dbd750_b043_4950_b08c_ef027ef51ef8.slice/crio-a2185535209ae67db5cef68943068fd06add7c9795d3922978fa1511e394cb5c WatchSource:0}: Error finding container a2185535209ae67db5cef68943068fd06add7c9795d3922978fa1511e394cb5c: Status 404 returned error can't find the container with id a2185535209ae67db5cef68943068fd06add7c9795d3922978fa1511e394cb5c
Nov 02 12:23:19.493260 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:19.493208    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"]
Nov 02 12:23:19.792946 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:19.792916    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"]
Nov 02 12:23:19.797206 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:19.797186    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"
Nov 02 12:23:19.856326 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:19.856293    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"]
Nov 02 12:23:19.911012 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:19.910954    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-plugins-config-volume\") pod \"payload-processing-external-model-749cdcd88d-dm2dx\" (UID: \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\") " pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"
Nov 02 12:23:19.911268 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:19.911244    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-provider-credentials\") pod \"payload-processing-external-model-749cdcd88d-dm2dx\" (UID: \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\") " pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"
Nov 02 12:23:19.911385 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:19.911368    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-routing-overlay-volume\") pod \"payload-processing-external-model-749cdcd88d-dm2dx\" (UID: \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\") " pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"
Nov 02 12:23:19.911448 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:19.911406    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpdv7\" (UniqueName: \"kubernetes.io/projected/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-kube-api-access-rpdv7\") pod \"payload-processing-external-model-749cdcd88d-dm2dx\" (UID: \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\") " pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"
Nov 02 12:23:20.012809 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.012780    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-provider-credentials\") pod \"payload-processing-external-model-749cdcd88d-dm2dx\" (UID: \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\") " pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"
Nov 02 12:23:20.012999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.012885    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-routing-overlay-volume\") pod \"payload-processing-external-model-749cdcd88d-dm2dx\" (UID: \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\") " pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"
Nov 02 12:23:20.012999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.012920    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpdv7\" (UniqueName: \"kubernetes.io/projected/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-kube-api-access-rpdv7\") pod \"payload-processing-external-model-749cdcd88d-dm2dx\" (UID: \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\") " pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"
Nov 02 12:23:20.013161 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.013141    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-plugins-config-volume\") pod \"payload-processing-external-model-749cdcd88d-dm2dx\" (UID: \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\") " pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"
Nov 02 12:23:20.013698 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.013672    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-routing-overlay-volume\") pod \"payload-processing-external-model-749cdcd88d-dm2dx\" (UID: \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\") " pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"
Nov 02 12:23:20.013824 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.013806    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-plugins-config-volume\") pod \"payload-processing-external-model-749cdcd88d-dm2dx\" (UID: \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\") " pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"
Nov 02 12:23:20.015339 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.015321    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-provider-credentials\") pod \"payload-processing-external-model-749cdcd88d-dm2dx\" (UID: \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\") " pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"
Nov 02 12:23:20.074756 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.074697    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpdv7\" (UniqueName: \"kubernetes.io/projected/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-kube-api-access-rpdv7\") pod \"payload-processing-external-model-749cdcd88d-dm2dx\" (UID: \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\") " pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"
Nov 02 12:23:20.111462 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.111438    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"
Nov 02 12:23:20.261303 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.261268    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7" event={"ID":"e9dbd750-b043-4950-b08c-ef027ef51ef8","Type":"ContainerStarted","Data":"3e49e0909eb4d7496ec5d61738bae6adc52c615918f4c50df8058ad2f92f245b"}
Nov 02 12:23:20.261464 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.261323    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7" event={"ID":"e9dbd750-b043-4950-b08c-ef027ef51ef8","Type":"ContainerStarted","Data":"a2185535209ae67db5cef68943068fd06add7c9795d3922978fa1511e394cb5c"}
Nov 02 12:23:20.343335 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.343262    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"]
Nov 02 12:23:20.348633 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.348612    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.384901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.384850    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7" podStartSLOduration=2.384837523 podStartE2EDuration="2.384837523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:23:20.382837136 +0000 UTC m=+1628.946545544" watchObservedRunningTime="2026-11-02 12:23:20.384837523 +0000 UTC m=+1628.948545935"
Nov 02 12:23:20.408740 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.408711    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"]
Nov 02 12:23:20.499081 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:23:20.499054    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70ff9e4_f0f1_4341_b4f6_dc67961c2255.slice/crio-197af3768df794de238e57b01357465506adf1b8723beb8de47107de0276b6e0 WatchSource:0}: Error finding container 197af3768df794de238e57b01357465506adf1b8723beb8de47107de0276b6e0: Status 404 returned error can't find the container with id 197af3768df794de238e57b01357465506adf1b8723beb8de47107de0276b6e0
Nov 02 12:23:20.499861 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.499836    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"]
Nov 02 12:23:20.518658 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.518634    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb51d042-8930-45f9-89c9-fe9ad351df24-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.518789 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.518673    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnc65\" (UniqueName: \"kubernetes.io/projected/eb51d042-8930-45f9-89c9-fe9ad351df24-kube-api-access-tnc65\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.518789 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.518707    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.518904 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.518871    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.519041 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.519013    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.519103 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.519076    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.519150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.519138    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.519225 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.519209    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.519344 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.519321    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.620547 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.620479    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.620547 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.620518    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.620724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.620579    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.620724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.620612    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.620868 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.620726    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb51d042-8930-45f9-89c9-fe9ad351df24-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.620868 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.620758    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnc65\" (UniqueName: \"kubernetes.io/projected/eb51d042-8930-45f9-89c9-fe9ad351df24-kube-api-access-tnc65\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.620868 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.620789    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.620868 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.620860    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.621111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.620920    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.621111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.620943    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.621111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.621011    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.621296 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.621270    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.621349 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.621289    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.621576 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.621557    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb51d042-8930-45f9-89c9-fe9ad351df24-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.622900 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.622880    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.623008 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.622971    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.682610 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.682578    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.682721 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.682695    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnc65\" (UniqueName: \"kubernetes.io/projected/eb51d042-8930-45f9-89c9-fe9ad351df24-kube-api-access-tnc65\") pod \"maas-default-gateway-openshift-default-6dcf757899-lwhz4\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:20.964242 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:20.964154    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:21.270470 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:21.270431    4464 generic.go:356] "Generic (PLEG): container finished" podID="e9dbd750-b043-4950-b08c-ef027ef51ef8" containerID="3e49e0909eb4d7496ec5d61738bae6adc52c615918f4c50df8058ad2f92f245b" exitCode=0
Nov 02 12:23:21.270638 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:21.270506    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7" event={"ID":"e9dbd750-b043-4950-b08c-ef027ef51ef8","Type":"ContainerDied","Data":"3e49e0909eb4d7496ec5d61738bae6adc52c615918f4c50df8058ad2f92f245b"}
Nov 02 12:23:21.272372 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:21.272349    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx" event={"ID":"d70ff9e4-f0f1-4341-b4f6-dc67961c2255","Type":"ContainerStarted","Data":"c5e10721fcc1714fc4c8acad4361fe5161bc349ccc20631b50e44a4e1db8ee02"}
Nov 02 12:23:21.272482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:21.272389    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx" event={"ID":"d70ff9e4-f0f1-4341-b4f6-dc67961c2255","Type":"ContainerStarted","Data":"197af3768df794de238e57b01357465506adf1b8723beb8de47107de0276b6e0"}
Nov 02 12:23:21.272482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:21.272410    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"
Nov 02 12:23:21.399607 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:23:21.398955    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb51d042_8930_45f9_89c9_fe9ad351df24.slice/crio-afb4502e422822f17bf0bfd3570e5a8aef741458dad016090de0ea0824f93b4e WatchSource:0}: Error finding container afb4502e422822f17bf0bfd3570e5a8aef741458dad016090de0ea0824f93b4e: Status 404 returned error can't find the container with id afb4502e422822f17bf0bfd3570e5a8aef741458dad016090de0ea0824f93b4e
Nov 02 12:23:21.399607 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:21.399055    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"]
Nov 02 12:23:21.402209 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:21.402171    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:23:21.402325 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:21.402246    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:23:21.402325 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:21.402287    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:23:21.513586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:21.513539    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx" podStartSLOduration=2.513523649 podStartE2EDuration="2.513523649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:23:21.507010639 +0000 UTC m=+1630.070719048" watchObservedRunningTime="2026-11-02 12:23:21.513523649 +0000 UTC m=+1630.077232069"
Nov 02 12:23:22.293565 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:22.293522    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4" event={"ID":"eb51d042-8930-45f9-89c9-fe9ad351df24","Type":"ContainerStarted","Data":"9cf4dc362091b40edd23666bb3978ed460ed494b6e14a13c5b60cdb2eb37a3e3"}
Nov 02 12:23:22.294076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:22.293593    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4" event={"ID":"eb51d042-8930-45f9-89c9-fe9ad351df24","Type":"ContainerStarted","Data":"afb4502e422822f17bf0bfd3570e5a8aef741458dad016090de0ea0824f93b4e"}
Nov 02 12:23:22.497026 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:22.496968    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"
Nov 02 12:23:22.547715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:22.547647    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9dbd750-b043-4950-b08c-ef027ef51ef8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dbd750-b043-4950-b08c-ef027ef51ef8-maas-api-openshift-ca\") pod \"e9dbd750-b043-4950-b08c-ef027ef51ef8\" (UID: \"e9dbd750-b043-4950-b08c-ef027ef51ef8\") "
Nov 02 12:23:22.547715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:22.547693    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9dbd750-b043-4950-b08c-ef027ef51ef8-kube-api-access-lfkz9\" (UniqueName: \"kubernetes.io/projected/e9dbd750-b043-4950-b08c-ef027ef51ef8-kube-api-access-lfkz9\") pod \"e9dbd750-b043-4950-b08c-ef027ef51ef8\" (UID: \"e9dbd750-b043-4950-b08c-ef027ef51ef8\") "
Nov 02 12:23:22.547944 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:22.547792    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9dbd750-b043-4950-b08c-ef027ef51ef8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dbd750-b043-4950-b08c-ef027ef51ef8-maas-api-xks-ca\") pod \"e9dbd750-b043-4950-b08c-ef027ef51ef8\" (UID: \"e9dbd750-b043-4950-b08c-ef027ef51ef8\") "
Nov 02 12:23:22.548104 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:22.548082    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9dbd750-b043-4950-b08c-ef027ef51ef8-maas-api-openshift-ca" pod "e9dbd750-b043-4950-b08c-ef027ef51ef8" (UID: "e9dbd750-b043-4950-b08c-ef027ef51ef8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:23:22.549796 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:22.549772    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9dbd750-b043-4950-b08c-ef027ef51ef8-maas-api-xks-ca" pod "e9dbd750-b043-4950-b08c-ef027ef51ef8" (UID: "e9dbd750-b043-4950-b08c-ef027ef51ef8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:23:22.549889 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:22.549808    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9dbd750-b043-4950-b08c-ef027ef51ef8-kube-api-access-lfkz9" pod "e9dbd750-b043-4950-b08c-ef027ef51ef8" (UID: "e9dbd750-b043-4950-b08c-ef027ef51ef8"). InnerVolumeSpecName "kube-api-access-lfkz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:23:22.625001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:22.624930    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4" podStartSLOduration=2.624915193 podStartE2EDuration="2.624915193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:23:22.418398785 +0000 UTC m=+1630.982107212" watchObservedRunningTime="2026-11-02 12:23:22.624915193 +0000 UTC m=+1631.188623601"
Nov 02 12:23:22.649598 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:22.649566    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9dbd750-b043-4950-b08c-ef027ef51ef8-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:23:22.649598 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:22.649594    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfkz9\" (UniqueName: \"kubernetes.io/projected/e9dbd750-b043-4950-b08c-ef027ef51ef8-kube-api-access-lfkz9\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:23:22.649837 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:22.649605    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9dbd750-b043-4950-b08c-ef027ef51ef8-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:23:22.983998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:22.982132    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:22.986090 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:22.986060    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lwhz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 02 12:23:22.986182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:22.986117    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4" podUID="eb51d042-8930-45f9-89c9-fe9ad351df24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 02 12:23:23.302302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:23.302223    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"
Nov 02 12:23:23.302302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:23.302260    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7" event={"ID":"e9dbd750-b043-4950-b08c-ef027ef51ef8","Type":"ContainerDied","Data":"a2185535209ae67db5cef68943068fd06add7c9795d3922978fa1511e394cb5c"}
Nov 02 12:23:23.302749 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:23.302307    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2185535209ae67db5cef68943068fd06add7c9795d3922978fa1511e394cb5c"
Nov 02 12:23:23.981000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:23.972388    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lwhz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 02 12:23:23.981000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:23.972450    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4" podUID="eb51d042-8930-45f9-89c9-fe9ad351df24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 02 12:23:24.144274 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:24.144038    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"]
Nov 02 12:23:24.204116 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:24.159771    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dabc0d1-cjzsv"]
Nov 02 12:23:24.983427 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:24.983390    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lwhz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 02 12:23:24.984031 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:24.983998    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4" podUID="eb51d042-8930-45f9-89c9-fe9ad351df24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 02 12:23:25.591295 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:25.591258    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9caa8f05-7487-4eec-87fb-cdb3b85d5f96" path="/var/lib/kubelet/pods/9caa8f05-7487-4eec-87fb-cdb3b85d5f96/volumes"
Nov 02 12:23:25.980182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:25.978366    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lwhz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 02 12:23:25.980182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:25.978424    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4" podUID="eb51d042-8930-45f9-89c9-fe9ad351df24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 02 12:23:26.976110 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:26.976071    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lwhz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 02 12:23:26.976684 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:26.976653    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4" podUID="eb51d042-8930-45f9-89c9-fe9ad351df24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 02 12:23:27.630286 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:27.630241    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g" podUID="f1252145-23ce-4c2c-bc8f-26fd85eb14f9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:23:27.630507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:27.630327    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4" podUID="9c76562c-76e1-4367-a648-59f93a59e7ba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:23:27.964934 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:27.964829    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lwhz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 02 12:23:27.964934 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:27.964897    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4" podUID="eb51d042-8930-45f9-89c9-fe9ad351df24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 02 12:23:28.294026 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:23:28.293957    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:23:28.965677 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:28.965627    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lwhz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 02 12:23:28.965890 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:28.965699    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4" podUID="eb51d042-8930-45f9-89c9-fe9ad351df24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 02 12:23:30.331646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.331230    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"
Nov 02 12:23:30.380111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.378884    4464 generic.go:356] "Generic (PLEG): container finished" podID="9c76562c-76e1-4367-a648-59f93a59e7ba" containerID="96812a29a69c3e2bbf0e80ad7ce3056647a3cf88c3c73ec476ad25be501f5761" exitCode=0
Nov 02 12:23:30.380111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.379215    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4" event={"ID":"9c76562c-76e1-4367-a648-59f93a59e7ba","Type":"ContainerDied","Data":"96812a29a69c3e2bbf0e80ad7ce3056647a3cf88c3c73ec476ad25be501f5761"}
Nov 02 12:23:30.380111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.379273    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4" event={"ID":"9c76562c-76e1-4367-a648-59f93a59e7ba","Type":"ContainerDied","Data":"a7730ecd9ed4b1b8137b92a0f0031fd15cba099c469129b729fa78910cbd08ea"}
Nov 02 12:23:30.380111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.379305    4464 scope.go:122] "RemoveContainer" containerID="96812a29a69c3e2bbf0e80ad7ce3056647a3cf88c3c73ec476ad25be501f5761"
Nov 02 12:23:30.380111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.379476    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"
Nov 02 12:23:30.383771 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.383745    4464 generic.go:356] "Generic (PLEG): container finished" podID="f1252145-23ce-4c2c-bc8f-26fd85eb14f9" containerID="3b14d8086fa48f12f43942c6ae66f8a36e74d7d11fe0c7aedde9014ad32ade0d" exitCode=0
Nov 02 12:23:30.383948 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.383823    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g" event={"ID":"f1252145-23ce-4c2c-bc8f-26fd85eb14f9","Type":"ContainerDied","Data":"3b14d8086fa48f12f43942c6ae66f8a36e74d7d11fe0c7aedde9014ad32ade0d"}
Nov 02 12:23:30.423453 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.423248    4464 scope.go:122] "RemoveContainer" containerID="96812a29a69c3e2bbf0e80ad7ce3056647a3cf88c3c73ec476ad25be501f5761"
Nov 02 12:23:30.423657 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:23:30.423624    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96812a29a69c3e2bbf0e80ad7ce3056647a3cf88c3c73ec476ad25be501f5761\": container with ID starting with 96812a29a69c3e2bbf0e80ad7ce3056647a3cf88c3c73ec476ad25be501f5761 not found: ID does not exist" containerID="96812a29a69c3e2bbf0e80ad7ce3056647a3cf88c3c73ec476ad25be501f5761"
Nov 02 12:23:30.423742 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.423674    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96812a29a69c3e2bbf0e80ad7ce3056647a3cf88c3c73ec476ad25be501f5761"} err="failed to get container status \"96812a29a69c3e2bbf0e80ad7ce3056647a3cf88c3c73ec476ad25be501f5761\": rpc error: code = NotFound desc = could not find container \"96812a29a69c3e2bbf0e80ad7ce3056647a3cf88c3c73ec476ad25be501f5761\": container with ID starting with 96812a29a69c3e2bbf0e80ad7ce3056647a3cf88c3c73ec476ad25be501f5761 not found: ID does not exist"
Nov 02 12:23:30.446426 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.445596    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c76562c-76e1-4367-a648-59f93a59e7ba-kube-api-access-f6fgr\" (UniqueName: \"kubernetes.io/projected/9c76562c-76e1-4367-a648-59f93a59e7ba-kube-api-access-f6fgr\") pod \"9c76562c-76e1-4367-a648-59f93a59e7ba\" (UID: \"9c76562c-76e1-4367-a648-59f93a59e7ba\") "
Nov 02 12:23:30.446426 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.445660    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c76562c-76e1-4367-a648-59f93a59e7ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c76562c-76e1-4367-a648-59f93a59e7ba-plugins-config-volume\") pod \"9c76562c-76e1-4367-a648-59f93a59e7ba\" (UID: \"9c76562c-76e1-4367-a648-59f93a59e7ba\") "
Nov 02 12:23:30.448002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.447952    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c76562c-76e1-4367-a648-59f93a59e7ba-plugins-config-volume" pod "9c76562c-76e1-4367-a648-59f93a59e7ba" (UID: "9c76562c-76e1-4367-a648-59f93a59e7ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:23:30.450467 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.450437    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c76562c-76e1-4367-a648-59f93a59e7ba-kube-api-access-f6fgr" pod "9c76562c-76e1-4367-a648-59f93a59e7ba" (UID: "9c76562c-76e1-4367-a648-59f93a59e7ba"). InnerVolumeSpecName "kube-api-access-f6fgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:23:30.475391 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.475362    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"
Nov 02 12:23:30.547254 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.547176    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1252145-23ce-4c2c-bc8f-26fd85eb14f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1252145-23ce-4c2c-bc8f-26fd85eb14f9-plugins-config-volume\") pod \"f1252145-23ce-4c2c-bc8f-26fd85eb14f9\" (UID: \"f1252145-23ce-4c2c-bc8f-26fd85eb14f9\") "
Nov 02 12:23:30.547407 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.547278    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1252145-23ce-4c2c-bc8f-26fd85eb14f9-kube-api-access-v6ms9\" (UniqueName: \"kubernetes.io/projected/f1252145-23ce-4c2c-bc8f-26fd85eb14f9-kube-api-access-v6ms9\") pod \"f1252145-23ce-4c2c-bc8f-26fd85eb14f9\" (UID: \"f1252145-23ce-4c2c-bc8f-26fd85eb14f9\") "
Nov 02 12:23:30.547643 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.547614    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1252145-23ce-4c2c-bc8f-26fd85eb14f9-plugins-config-volume" pod "f1252145-23ce-4c2c-bc8f-26fd85eb14f9" (UID: "f1252145-23ce-4c2c-bc8f-26fd85eb14f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:23:30.547718 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.547685    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6fgr\" (UniqueName: \"kubernetes.io/projected/9c76562c-76e1-4367-a648-59f93a59e7ba-kube-api-access-f6fgr\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:23:30.547718 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.547708    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c76562c-76e1-4367-a648-59f93a59e7ba-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:23:30.549348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.549321    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1252145-23ce-4c2c-bc8f-26fd85eb14f9-kube-api-access-v6ms9" pod "f1252145-23ce-4c2c-bc8f-26fd85eb14f9" (UID: "f1252145-23ce-4c2c-bc8f-26fd85eb14f9"). InnerVolumeSpecName "kube-api-access-v6ms9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:23:30.650019 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.649956    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6ms9\" (UniqueName: \"kubernetes.io/projected/f1252145-23ce-4c2c-bc8f-26fd85eb14f9-kube-api-access-v6ms9\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:23:30.650019 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.649998    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1252145-23ce-4c2c-bc8f-26fd85eb14f9-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:23:30.811070 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.810962    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"]
Nov 02 12:23:30.815003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:30.814958    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-qchj4"]
Nov 02 12:23:31.006158 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:31.006119    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lwhz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:23:31.006301 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:31.006172    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4" podUID="eb51d042-8930-45f9-89c9-fe9ad351df24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:23:31.146087 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:31.145967    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:31.146703 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:31.146680    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:31.147314 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:31.147295    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:23:31.408283 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:31.399688    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"
Nov 02 12:23:31.408283 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:31.399724    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g" event={"ID":"f1252145-23ce-4c2c-bc8f-26fd85eb14f9","Type":"ContainerDied","Data":"b5af3e7a1c87a47b3ec591af69090472b793a69176284d0cbbbbfe0ba2674e35"}
Nov 02 12:23:31.408283 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:31.399874    4464 scope.go:122] "RemoveContainer" containerID="3b14d8086fa48f12f43942c6ae66f8a36e74d7d11fe0c7aedde9014ad32ade0d"
Nov 02 12:23:31.586032 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:31.585992    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c76562c-76e1-4367-a648-59f93a59e7ba" path="/var/lib/kubelet/pods/9c76562c-76e1-4367-a648-59f93a59e7ba/volumes"
Nov 02 12:23:31.586300 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:31.586287    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"]
Nov 02 12:23:31.586344 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:31.586316    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-h6n4g"]
Nov 02 12:23:32.296597 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:32.296560    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx" podUID="d70ff9e4-f0f1-4341-b4f6-dc67961c2255" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:23:33.584704 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:33.584668    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1252145-23ce-4c2c-bc8f-26fd85eb14f9" path="/var/lib/kubelet/pods/f1252145-23ce-4c2c-bc8f-26fd85eb14f9/volumes"
Nov 02 12:23:34.051523 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:34.051424    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"]
Nov 02 12:23:34.099901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:34.099864    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fd29b28-mt8w7"]
Nov 02 12:23:35.460717 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.460684    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"]
Nov 02 12:23:35.461299 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.461285    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f1252145-23ce-4c2c-bc8f-26fd85eb14f9" containerName="payload-pre-processing"
Nov 02 12:23:35.461348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.461300    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1252145-23ce-4c2c-bc8f-26fd85eb14f9" containerName="payload-pre-processing"
Nov 02 12:23:35.461348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.461327    4464 cpu_manager.go:414] "containerMap: removing container" podUID="9c76562c-76e1-4367-a648-59f93a59e7ba" containerName="payload-processing"
Nov 02 12:23:35.461348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.461332    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c76562c-76e1-4367-a648-59f93a59e7ba" containerName="payload-processing"
Nov 02 12:23:35.461443 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.461359    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e9dbd750-b043-4950-b08c-ef027ef51ef8" containerName="revoke-keys"
Nov 02 12:23:35.461443 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.461364    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9dbd750-b043-4950-b08c-ef027ef51ef8" containerName="revoke-keys"
Nov 02 12:23:35.461504 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.461445    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c76562c-76e1-4367-a648-59f93a59e7ba" containerName="payload-processing"
Nov 02 12:23:35.461504 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.461459    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9dbd750-b043-4950-b08c-ef027ef51ef8" containerName="revoke-keys"
Nov 02 12:23:35.461504 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.461467    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1252145-23ce-4c2c-bc8f-26fd85eb14f9" containerName="payload-pre-processing"
Nov 02 12:23:35.464964 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.464945    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.511192 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.511164    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"]
Nov 02 12:23:35.515288 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.515269    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.534374 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.534350    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.534481 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.534391    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.534573 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.534546    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.534641 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.534586    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.534641 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.534616    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896e3cac-30cf-492b-a20a-cec437dc6a05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.534762 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.534706    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.534810 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.534774    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.534880 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.534861    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.534943 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.534926    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsgjd\" (UniqueName: \"kubernetes.io/projected/896e3cac-30cf-492b-a20a-cec437dc6a05-kube-api-access-xsgjd\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.565196 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.565173    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"]
Nov 02 12:23:35.582968 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.582945    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16f72bca-a91b-41d8-b3e7-764c9e53ad5f" path="/var/lib/kubelet/pods/16f72bca-a91b-41d8-b3e7-764c9e53ad5f/volumes"
Nov 02 12:23:35.619579 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.619547    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"]
Nov 02 12:23:35.635959 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.635931    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsgjd\" (UniqueName: \"kubernetes.io/projected/896e3cac-30cf-492b-a20a-cec437dc6a05-kube-api-access-xsgjd\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.636118 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.635985    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.636118 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.636036    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.636118 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.636063    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.636118 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.636089    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.636362 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.636322    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.636421 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.636398    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.636527 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.636505    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.636648 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.636562    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.636648 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.636595    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.636648 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.636622    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.636810 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.636645    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896e3cac-30cf-492b-a20a-cec437dc6a05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.636810 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.636711    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.636918 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.636820    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.636918 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.636900    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.637564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.636968    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.637564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.637030    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.637564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.637071    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.637564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.637100    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6644t\" (UniqueName: \"kubernetes.io/projected/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-kube-api-access-6644t\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.637564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.637209    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.637841 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.637691    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896e3cac-30cf-492b-a20a-cec437dc6a05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.637841 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.637754    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.637950 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.637905    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.640078 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.640046    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.640200 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.640106    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.697949 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.697912    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsgjd\" (UniqueName: \"kubernetes.io/projected/896e3cac-30cf-492b-a20a-cec437dc6a05-kube-api-access-xsgjd\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.697949 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.697935    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-62g8c\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.738667 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.738629    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.738808 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.738692    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.738808 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.738746    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.738919 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.738808    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.738919 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.738832    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.738919 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.738860    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.739091 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.739014    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6644t\" (UniqueName: \"kubernetes.io/projected/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-kube-api-access-6644t\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.739153 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.739088    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.739153 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.739141    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.739256 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.739230    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.739319 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.739267    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.739553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.739528    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.739661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.739535    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.739767 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.739751    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.740856 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.740833    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.741004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.740969    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.778528 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.778506    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:35.800553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.800528    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.800661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.800621    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6644t\" (UniqueName: \"kubernetes.io/projected/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-kube-api-access-6644t\") pod \"maas-default-gateway-openshift-default-6dcf757899-jthbv\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:35.828873 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:35.828844    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:36.207018 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:23:36.206956    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896e3cac_30cf_492b_a20a_cec437dc6a05.slice/crio-fac98a43db353bccbc05f156e2e8d30c6a751686a188dff16da6a573ac48a0b8 WatchSource:0}: Error finding container fac98a43db353bccbc05f156e2e8d30c6a751686a188dff16da6a573ac48a0b8: Status 404 returned error can't find the container with id fac98a43db353bccbc05f156e2e8d30c6a751686a188dff16da6a573ac48a0b8
Nov 02 12:23:36.208763 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:36.208742    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"]
Nov 02 12:23:36.209863 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:36.209835    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:23:36.209940 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:36.209895    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:23:36.209940 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:36.209924    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:23:36.261372 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:36.261350    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"]
Nov 02 12:23:36.261676 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:23:36.261658    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890b9eeb_eb6b_4cc5_beff_222fcdc94d2f.slice/crio-383be9d637c7cdfc0f2e35c8ce386241d38177022e5ac5c64d075b0cc6c297e9 WatchSource:0}: Error finding container 383be9d637c7cdfc0f2e35c8ce386241d38177022e5ac5c64d075b0cc6c297e9: Status 404 returned error can't find the container with id 383be9d637c7cdfc0f2e35c8ce386241d38177022e5ac5c64d075b0cc6c297e9
Nov 02 12:23:36.265004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:36.264927    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:23:36.265273 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:36.265244    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:23:36.265452 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:36.265429    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:23:36.510719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:36.510524    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c" event={"ID":"896e3cac-30cf-492b-a20a-cec437dc6a05","Type":"ContainerStarted","Data":"419f3f75d7874891e9dd2af2149c76a2ce261aee46c0ece5e5f51d184547be5a"}
Nov 02 12:23:36.510719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:36.510643    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c" event={"ID":"896e3cac-30cf-492b-a20a-cec437dc6a05","Type":"ContainerStarted","Data":"fac98a43db353bccbc05f156e2e8d30c6a751686a188dff16da6a573ac48a0b8"}
Nov 02 12:23:36.512840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:36.512811    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv" event={"ID":"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f","Type":"ContainerStarted","Data":"f82b80bb5789034c6329a820dc78fb89a27d1b57da62eb9d50f37feb5629b1e5"}
Nov 02 12:23:36.513013 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:36.512862    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv" event={"ID":"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f","Type":"ContainerStarted","Data":"383be9d637c7cdfc0f2e35c8ce386241d38177022e5ac5c64d075b0cc6c297e9"}
Nov 02 12:23:36.636683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:36.636628    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c" podStartSLOduration=1.636612095 podStartE2EDuration="1.636612095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:23:36.631664107 +0000 UTC m=+1645.195372517" watchObservedRunningTime="2026-11-02 12:23:36.636612095 +0000 UTC m=+1645.200320504"
Nov 02 12:23:36.757520 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:36.757464    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv" podStartSLOduration=1.757449963 podStartE2EDuration="1.757449963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:23:36.753028603 +0000 UTC m=+1645.316737013" watchObservedRunningTime="2026-11-02 12:23:36.757449963 +0000 UTC m=+1645.321158450"
Nov 02 12:23:36.779040 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:36.778940    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:36.829881 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:36.829840    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:37.794000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:37.793106    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-62g8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 12:23:37.794000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:37.793169    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c" podUID="896e3cac-30cf-492b-a20a-cec437dc6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 12:23:37.831888 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:37.831852    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jthbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:23:37.832085 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:37.831904    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv" podUID="890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 12:23:38.779999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:38.779343    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-62g8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 12:23:38.779999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:38.779403    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c" podUID="896e3cac-30cf-492b-a20a-cec437dc6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 12:23:38.829997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:38.829766    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jthbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:23:38.829997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:38.829854    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv" podUID="890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 12:23:39.779376 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:39.779338    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-62g8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 12:23:39.779845 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:39.779816    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c" podUID="896e3cac-30cf-492b-a20a-cec437dc6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 12:23:39.829998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:39.829949    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jthbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:23:39.830171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:39.830024    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv" podUID="890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 12:23:40.783552 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:40.783513    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-62g8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 12:23:40.783888 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:40.783840    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c" podUID="896e3cac-30cf-492b-a20a-cec437dc6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 12:23:40.829592 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:40.829560    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jthbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:23:40.829759 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:40.829607    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv" podUID="890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 12:23:41.782000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:41.781316    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-62g8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 12:23:41.782000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:41.781374    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c" podUID="896e3cac-30cf-492b-a20a-cec437dc6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 12:23:41.829710 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:41.829680    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jthbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:23:41.829872 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:41.829730    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv" podUID="890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 12:23:42.300120 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:42.300071    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx" podUID="d70ff9e4-f0f1-4341-b4f6-dc67961c2255" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:23:42.779331 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:42.779293    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-62g8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 12:23:42.779507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:42.779361    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c" podUID="896e3cac-30cf-492b-a20a-cec437dc6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 12:23:42.830154 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:42.830122    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jthbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:23:42.830513 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:42.830174    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv" podUID="890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 12:23:43.778960 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:43.778920    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-62g8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 12:23:43.779142 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:43.779010    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c" podUID="896e3cac-30cf-492b-a20a-cec437dc6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 12:23:43.829658 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:43.829623    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jthbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:23:43.829797 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:43.829681    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv" podUID="890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 02 12:23:44.475290 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:44.475255    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/models-as-a-service_payload-processing-external-model-749cdcd88d-dm2dx_d70ff9e4-f0f1-4341-b4f6-dc67961c2255/payload-processing/0.log"
Nov 02 12:23:44.498735 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:44.498702    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:23:44.693000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:44.692950    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/models-as-a-service_payload-processing-external-model-866d944846-nnbj2_8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac/payload-processing/0.log"
Nov 02 12:23:44.779510 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:44.779413    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-62g8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 02 12:23:44.779826 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:44.779789    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c" podUID="896e3cac-30cf-492b-a20a-cec437dc6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 02 12:23:45.873272 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:45.873191    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jthbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:23:45.874038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:45.873995    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv" podUID="890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:23:46.696539 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:46.696503    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:46.697065 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:46.697026    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:46.697967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:46.697944    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:23:46.719783 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:46.719753    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:46.721905 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:46.721878    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:46.722693 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:46.722670    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:23:50.143581 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.143544    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"]
Nov 02 12:23:50.148364 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.148341    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:23:50.205688 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.205658    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"]
Nov 02 12:23:50.251592 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.251563    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-50db61f22b2e\""
Nov 02 12:23:50.251592 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.251583    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-4b8a830d91a3\""
Nov 02 12:23:50.251803 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.251706    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3a78b4\""
Nov 02 12:23:50.251803 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.251791    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"routing-overlay\""
Nov 02 12:23:50.252860 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.252841    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-4edd8a36d129\""
Nov 02 12:23:50.252860 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.252857    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-14e0f93131b7\""
Nov 02 12:23:50.253042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.252869    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"openshift-service-ca.crt\""
Nov 02 12:23:50.253042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.252922    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"payload-processing-external-model-e2e--a8ab13a3-dockercfg-wmcbx\""
Nov 02 12:23:50.253042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.252933    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"kube-root-ca.crt\""
Nov 02 12:23:50.293625 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.293599    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b309a8-faca-49c6-93da-9563221f653f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm\" (UID: \"14b309a8-faca-49c6-93da-9563221f653f\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:23:50.293790 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.293631    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8v6s\" (UniqueName: \"kubernetes.io/projected/14b309a8-faca-49c6-93da-9563221f653f-kube-api-access-h8v6s\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm\" (UID: \"14b309a8-faca-49c6-93da-9563221f653f\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:23:50.293790 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.293751    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/14b309a8-faca-49c6-93da-9563221f653f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm\" (UID: \"14b309a8-faca-49c6-93da-9563221f653f\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:23:50.293943 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.293924    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14b309a8-faca-49c6-93da-9563221f653f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm\" (UID: \"14b309a8-faca-49c6-93da-9563221f653f\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:23:50.395451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.395379    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/14b309a8-faca-49c6-93da-9563221f653f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm\" (UID: \"14b309a8-faca-49c6-93da-9563221f653f\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:23:50.395593 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.395537    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14b309a8-faca-49c6-93da-9563221f653f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm\" (UID: \"14b309a8-faca-49c6-93da-9563221f653f\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:23:50.395634 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.395614    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b309a8-faca-49c6-93da-9563221f653f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm\" (UID: \"14b309a8-faca-49c6-93da-9563221f653f\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:23:50.395669 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.395643    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8v6s\" (UniqueName: \"kubernetes.io/projected/14b309a8-faca-49c6-93da-9563221f653f-kube-api-access-h8v6s\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm\" (UID: \"14b309a8-faca-49c6-93da-9563221f653f\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:23:50.396282 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.396247    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14b309a8-faca-49c6-93da-9563221f653f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm\" (UID: \"14b309a8-faca-49c6-93da-9563221f653f\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:23:50.396424 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.396367    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b309a8-faca-49c6-93da-9563221f653f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm\" (UID: \"14b309a8-faca-49c6-93da-9563221f653f\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:23:50.398016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.397994    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/14b309a8-faca-49c6-93da-9563221f653f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm\" (UID: \"14b309a8-faca-49c6-93da-9563221f653f\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:23:50.459493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.459469    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8v6s\" (UniqueName: \"kubernetes.io/projected/14b309a8-faca-49c6-93da-9563221f653f-kube-api-access-h8v6s\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm\" (UID: \"14b309a8-faca-49c6-93da-9563221f653f\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:23:50.463403 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.463381    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:23:50.844525 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:23:50.844489    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b309a8_faca_49c6_93da_9563221f653f.slice/crio-19416a0cc701de7108b903b48dc19c45350363c63db9609ddfea190a2d362773 WatchSource:0}: Error finding container 19416a0cc701de7108b903b48dc19c45350363c63db9609ddfea190a2d362773: Status 404 returned error can't find the container with id 19416a0cc701de7108b903b48dc19c45350363c63db9609ddfea190a2d362773
Nov 02 12:23:50.845506 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:50.845477    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"]
Nov 02 12:23:51.642877 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:51.642840    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm" event={"ID":"14b309a8-faca-49c6-93da-9563221f653f","Type":"ContainerStarted","Data":"6c944cf139ecadb260ae12b9bbbed51d8ba07d4874413b990dc292817e98ef4d"}
Nov 02 12:23:51.643301 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:51.642902    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:23:51.643301 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:51.642918    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm" event={"ID":"14b309a8-faca-49c6-93da-9563221f653f","Type":"ContainerStarted","Data":"19416a0cc701de7108b903b48dc19c45350363c63db9609ddfea190a2d362773"}
Nov 02 12:23:51.764803 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:51.764750    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm" podStartSLOduration=1.7647311239999999 podStartE2EDuration="1.764731124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:23:51.760619503 +0000 UTC m=+1660.324327916" watchObservedRunningTime="2026-11-02 12:23:51.764731124 +0000 UTC m=+1660.328439533"
Nov 02 12:23:52.295204 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:52.295173    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx" podUID="d70ff9e4-f0f1-4341-b4f6-dc67961c2255" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:23:53.068879 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.068842    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"]
Nov 02 12:23:53.103254 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.103219    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a2c897-xrml8"]
Nov 02 12:23:53.497998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.497950    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"]
Nov 02 12:23:53.502551 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.502531    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"
Nov 02 12:23:53.560820 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.560791    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"]
Nov 02 12:23:53.584536 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.584499    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a2d46c4-eb23-4706-93d6-e9f3f4facd2a" path="/var/lib/kubelet/pods/4a2d46c4-eb23-4706-93d6-e9f3f4facd2a/volumes"
Nov 02 12:23:53.606876 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.606805    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"agc-cred-a7c8aaecea68\""
Nov 02 12:23:53.607053 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.607027    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6ac03f\""
Nov 02 12:23:53.607135 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.607066    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"openshift-service-ca.crt\""
Nov 02 12:23:53.607135 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.607067    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"kube-root-ca.crt\""
Nov 02 12:23:53.607135 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.607086    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"routing-overlay\""
Nov 02 12:23:53.607253 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.607148    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"payload-processing-external-model-e2e--75a83f02-dockercfg-fkh9m\""
Nov 02 12:23:53.643538 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.643505    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/271e7877-ab08-4f16-9947-5910eb6a29fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8\" (UID: \"271e7877-ab08-4f16-9947-5910eb6a29fb\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"
Nov 02 12:23:53.643689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.643560    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271e7877-ab08-4f16-9947-5910eb6a29fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8\" (UID: \"271e7877-ab08-4f16-9947-5910eb6a29fb\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"
Nov 02 12:23:53.643689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.643625    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8lpd\" (UniqueName: \"kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-kube-api-access-g8lpd\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8\" (UID: \"271e7877-ab08-4f16-9947-5910eb6a29fb\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"
Nov 02 12:23:53.643836 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.643814    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8\" (UID: \"271e7877-ab08-4f16-9947-5910eb6a29fb\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"
Nov 02 12:23:53.745117 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.745086    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/271e7877-ab08-4f16-9947-5910eb6a29fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8\" (UID: \"271e7877-ab08-4f16-9947-5910eb6a29fb\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"
Nov 02 12:23:53.745295 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.745143    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271e7877-ab08-4f16-9947-5910eb6a29fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8\" (UID: \"271e7877-ab08-4f16-9947-5910eb6a29fb\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"
Nov 02 12:23:53.745295 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.745174    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8lpd\" (UniqueName: \"kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-kube-api-access-g8lpd\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8\" (UID: \"271e7877-ab08-4f16-9947-5910eb6a29fb\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"
Nov 02 12:23:53.745381 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.745300    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8\" (UID: \"271e7877-ab08-4f16-9947-5910eb6a29fb\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"
Nov 02 12:23:53.745823 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.745799    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/271e7877-ab08-4f16-9947-5910eb6a29fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8\" (UID: \"271e7877-ab08-4f16-9947-5910eb6a29fb\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"
Nov 02 12:23:53.745901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.745881    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271e7877-ab08-4f16-9947-5910eb6a29fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8\" (UID: \"271e7877-ab08-4f16-9947-5910eb6a29fb\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"
Nov 02 12:23:53.747715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.747697    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8\" (UID: \"271e7877-ab08-4f16-9947-5910eb6a29fb\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"
Nov 02 12:23:53.842047 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:53.831537    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8lpd\" (UniqueName: \"kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-kube-api-access-g8lpd\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8\" (UID: \"271e7877-ab08-4f16-9947-5910eb6a29fb\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"
Nov 02 12:23:54.119288 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:54.119205    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"
Nov 02 12:23:54.502708 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:23:54.502678    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271e7877_ab08_4f16_9947_5910eb6a29fb.slice/crio-9f6904060a8d8b137a8601ac112a4febf0fc55369650516a9e9840a4f90705d1 WatchSource:0}: Error finding container 9f6904060a8d8b137a8601ac112a4febf0fc55369650516a9e9840a4f90705d1: Status 404 returned error can't find the container with id 9f6904060a8d8b137a8601ac112a4febf0fc55369650516a9e9840a4f90705d1
Nov 02 12:23:54.503189 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:54.503166    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"]
Nov 02 12:23:54.667228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:54.667195    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8" event={"ID":"271e7877-ab08-4f16-9947-5910eb6a29fb","Type":"ContainerStarted","Data":"d1003463503917ae07da1bef2a1a2c72fa1bbb24df1696e676e9526e7bf5aea9"}
Nov 02 12:23:54.667380 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:54.667256    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"
Nov 02 12:23:54.667380 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:54.667276    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8" event={"ID":"271e7877-ab08-4f16-9947-5910eb6a29fb","Type":"ContainerStarted","Data":"9f6904060a8d8b137a8601ac112a4febf0fc55369650516a9e9840a4f90705d1"}
Nov 02 12:23:54.786168 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:54.786044    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8" podStartSLOduration=1.7860301669999998 podStartE2EDuration="1.786030167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:23:54.785414705 +0000 UTC m=+1663.349123115" watchObservedRunningTime="2026-11-02 12:23:54.786030167 +0000 UTC m=+1663.349738576"
Nov 02 12:23:55.559544 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:55.559509    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"]
Nov 02 12:23:55.562256 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:55.562218    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:23:55.790145 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:23:55.790110    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-a7c8aaecea68: secret "agc-cred-a7c8aaecea68" not found
Nov 02 12:23:55.790145 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:23:55.790140    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8: secret "agc-cred-a7c8aaecea68" not found
Nov 02 12:23:55.790346 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:23:55.790215    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-provider-credentials podName:271e7877-ab08-4f16-9947-5910eb6a29fb nodeName:}" failed. No retries permitted until 2026-11-02 12:23:56.290191837 +0000 UTC m=+1664.853900238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8" (UID: "271e7877-ab08-4f16-9947-5910eb6a29fb") : secret "agc-cred-a7c8aaecea68" not found
Nov 02 12:23:56.302478 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:23:56.302443    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-a7c8aaecea68: secret "agc-cred-a7c8aaecea68" not found
Nov 02 12:23:56.302478 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:23:56.302470    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8: secret "agc-cred-a7c8aaecea68" not found
Nov 02 12:23:56.302712 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:23:56.302519    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-provider-credentials podName:271e7877-ab08-4f16-9947-5910eb6a29fb nodeName:}" failed. No retries permitted until 2026-11-02 12:23:57.302507385 +0000 UTC m=+1665.866215771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8" (UID: "271e7877-ab08-4f16-9947-5910eb6a29fb") : secret "agc-cred-a7c8aaecea68" not found
Nov 02 12:23:57.124926 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:23:57.124894    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"]
Nov 02 12:23:57.335994 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:23:57.330496    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-a7c8aaecea68: secret "agc-cred-a7c8aaecea68" not found
Nov 02 12:23:57.335994 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:23:57.330532    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8: secret "agc-cred-a7c8aaecea68" not found
Nov 02 12:23:57.335994 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:23:57.330606    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-provider-credentials podName:271e7877-ab08-4f16-9947-5910eb6a29fb nodeName:}" failed. No retries permitted until 2026-11-02 12:23:59.330584805 +0000 UTC m=+1667.894293216 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8" (UID: "271e7877-ab08-4f16-9947-5910eb6a29fb") : secret "agc-cred-a7c8aaecea68" not found
Nov 02 12:23:58.292381 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:23:58.292341    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:23:59.350351 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:23:59.350321    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-a7c8aaecea68: secret "agc-cred-a7c8aaecea68" not found
Nov 02 12:23:59.350351 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:23:59.350343    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8: secret "agc-cred-a7c8aaecea68" not found
Nov 02 12:23:59.350804 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:23:59.350394    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-provider-credentials podName:271e7877-ab08-4f16-9947-5910eb6a29fb nodeName:}" failed. No retries permitted until 2026-11-02 12:24:03.350380208 +0000 UTC m=+1671.914088596 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8" (UID: "271e7877-ab08-4f16-9947-5910eb6a29fb") : secret "agc-cred-a7c8aaecea68" not found
Nov 02 12:24:00.223607 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.223577    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"]
Nov 02 12:24:00.227862 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.227846    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.281296 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.281267    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6a432197-openshift-default-dockercfg-n865n\""
Nov 02 12:24:00.338665 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.338630    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"]
Nov 02 12:24:00.362856 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.362819    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-token\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.363359 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.362930    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-credential-socket\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.363359 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.363135    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-podinfo\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.363359 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.363169    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-data\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.363359 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.363206    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-workload-socket\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.363359 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.363340    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e834d8c0-89ce-4797-b500-fd61f84ad58e-istiod-ca-cert\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.363635 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.363376    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwnww\" (UniqueName: \"kubernetes.io/projected/e834d8c0-89ce-4797-b500-fd61f84ad58e-kube-api-access-bwnww\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.363635 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.363511    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-envoy\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.363635 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.363590    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-workload-certs\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.465025 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.464987    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-podinfo\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.465025 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.465025    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-data\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.465260 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.465059    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-workload-socket\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.465260 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.465093    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e834d8c0-89ce-4797-b500-fd61f84ad58e-istiod-ca-cert\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.465260 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.465120    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwnww\" (UniqueName: \"kubernetes.io/projected/e834d8c0-89ce-4797-b500-fd61f84ad58e-kube-api-access-bwnww\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.465260 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.465184    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-envoy\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.465491 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.465257    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-workload-certs\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.465491 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.465367    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-token\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.465491 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.465403    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-credential-socket\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.465639 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.465528    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-workload-socket\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.465639 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.465621    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-workload-certs\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.465807 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.465779    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-data\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.465912 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.465837    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-credential-socket\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.465951 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.465914    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e834d8c0-89ce-4797-b500-fd61f84ad58e-istiod-ca-cert\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.467674 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.467650    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-podinfo\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.467780 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.467695    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-envoy\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.527046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.526960    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-token\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.527206 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.527186    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwnww\" (UniqueName: \"kubernetes.io/projected/e834d8c0-89ce-4797-b500-fd61f84ad58e-kube-api-access-bwnww\") pod \"e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") " pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.542614 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.542590    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:00.559762 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.559738    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm" podUID="14b309a8-faca-49c6-93da-9563221f653f" containerName="payload-processing" containerID="cri-o://6c944cf139ecadb260ae12b9bbbed51d8ba07d4874413b990dc292817e98ef4d" gracePeriod=25
Nov 02 12:24:00.715402 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.715370    4464 generic.go:356] "Generic (PLEG): container finished" podID="14b309a8-faca-49c6-93da-9563221f653f" containerID="6c944cf139ecadb260ae12b9bbbed51d8ba07d4874413b990dc292817e98ef4d" exitCode=0
Nov 02 12:24:00.715567 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.715442    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm" event={"ID":"14b309a8-faca-49c6-93da-9563221f653f","Type":"ContainerDied","Data":"6c944cf139ecadb260ae12b9bbbed51d8ba07d4874413b990dc292817e98ef4d"}
Nov 02 12:24:00.828632 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.828598    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:24:00.973100 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.973073    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b309a8-faca-49c6-93da-9563221f653f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/14b309a8-faca-49c6-93da-9563221f653f-provider-credentials\") pod \"14b309a8-faca-49c6-93da-9563221f653f\" (UID: \"14b309a8-faca-49c6-93da-9563221f653f\") "
Nov 02 12:24:00.973221 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.973157    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14b309a8-faca-49c6-93da-9563221f653f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14b309a8-faca-49c6-93da-9563221f653f-routing-overlay-volume\") pod \"14b309a8-faca-49c6-93da-9563221f653f\" (UID: \"14b309a8-faca-49c6-93da-9563221f653f\") "
Nov 02 12:24:00.973375 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.973358    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b309a8-faca-49c6-93da-9563221f653f-kube-api-access-h8v6s\" (UniqueName: \"kubernetes.io/projected/14b309a8-faca-49c6-93da-9563221f653f-kube-api-access-h8v6s\") pod \"14b309a8-faca-49c6-93da-9563221f653f\" (UID: \"14b309a8-faca-49c6-93da-9563221f653f\") "
Nov 02 12:24:00.973441 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.973409    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14b309a8-faca-49c6-93da-9563221f653f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b309a8-faca-49c6-93da-9563221f653f-plugins-config-volume\") pod \"14b309a8-faca-49c6-93da-9563221f653f\" (UID: \"14b309a8-faca-49c6-93da-9563221f653f\") "
Nov 02 12:24:00.974302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.974272    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b309a8-faca-49c6-93da-9563221f653f-routing-overlay-volume" pod "14b309a8-faca-49c6-93da-9563221f653f" (UID: "14b309a8-faca-49c6-93da-9563221f653f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:24:00.976408 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:24:00.976382    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode834d8c0_89ce_4797_b500_fd61f84ad58e.slice/crio-88769a8185a030b5d19a27a71a9f64dddd2e370c7e96b51e57f6f21fc49366ec WatchSource:0}: Error finding container 88769a8185a030b5d19a27a71a9f64dddd2e370c7e96b51e57f6f21fc49366ec: Status 404 returned error can't find the container with id 88769a8185a030b5d19a27a71a9f64dddd2e370c7e96b51e57f6f21fc49366ec
Nov 02 12:24:00.978732 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.978699    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:24:00.978839 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.978762    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:24:00.978839 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.978792    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:24:00.979928 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.979905    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"]
Nov 02 12:24:00.980196 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.980172    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b309a8-faca-49c6-93da-9563221f653f-plugins-config-volume" pod "14b309a8-faca-49c6-93da-9563221f653f" (UID: "14b309a8-faca-49c6-93da-9563221f653f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:24:00.980583 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.980556    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b309a8-faca-49c6-93da-9563221f653f-provider-credentials" pod "14b309a8-faca-49c6-93da-9563221f653f" (UID: "14b309a8-faca-49c6-93da-9563221f653f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:24:00.981865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:00.981843    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b309a8-faca-49c6-93da-9563221f653f-kube-api-access-h8v6s" pod "14b309a8-faca-49c6-93da-9563221f653f" (UID: "14b309a8-faca-49c6-93da-9563221f653f"). InnerVolumeSpecName "kube-api-access-h8v6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:24:01.093298 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:01.084294    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8v6s\" (UniqueName: \"kubernetes.io/projected/14b309a8-faca-49c6-93da-9563221f653f-kube-api-access-h8v6s\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:01.093298 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:01.084329    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b309a8-faca-49c6-93da-9563221f653f-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:01.093298 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:01.084348    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/14b309a8-faca-49c6-93da-9563221f653f-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:01.093298 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:01.084362    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14b309a8-faca-49c6-93da-9563221f653f-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:01.724804 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:01.724751    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d" event={"ID":"e834d8c0-89ce-4797-b500-fd61f84ad58e","Type":"ContainerStarted","Data":"bd2ec375f5adc3f7d24a3d6465bda54519f28cb2e8a1c1505170805cff48a648"}
Nov 02 12:24:01.725260 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:01.724824    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d" event={"ID":"e834d8c0-89ce-4797-b500-fd61f84ad58e","Type":"ContainerStarted","Data":"88769a8185a030b5d19a27a71a9f64dddd2e370c7e96b51e57f6f21fc49366ec"}
Nov 02 12:24:01.726451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:01.726425    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm" event={"ID":"14b309a8-faca-49c6-93da-9563221f653f","Type":"ContainerDied","Data":"19416a0cc701de7108b903b48dc19c45350363c63db9609ddfea190a2d362773"}
Nov 02 12:24:01.726592 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:01.726456    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"
Nov 02 12:24:01.726592 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:01.726473    4464 scope.go:122] "RemoveContainer" containerID="6c944cf139ecadb260ae12b9bbbed51d8ba07d4874413b990dc292817e98ef4d"
Nov 02 12:24:01.854517 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:01.854467    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d" podStartSLOduration=1.8544521440000001 podStartE2EDuration="1.854452144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:24:01.853356866 +0000 UTC m=+1670.417065285" watchObservedRunningTime="2026-11-02 12:24:01.854452144 +0000 UTC m=+1670.418160553"
Nov 02 12:24:01.988430 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:01.988338    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"]
Nov 02 12:24:01.993293 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:01.993264    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-777dxrgcm"]
Nov 02 12:24:02.125827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.125782    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8" podUID="271e7877-ab08-4f16-9947-5910eb6a29fb" containerName="payload-processing" containerID="cri-o://d1003463503917ae07da1bef2a1a2c72fa1bbb24df1696e676e9526e7bf5aea9" gracePeriod=25
Nov 02 12:24:02.295478 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.295443    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx" podUID="d70ff9e4-f0f1-4341-b4f6-dc67961c2255" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:24:02.484000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.483188    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"
Nov 02 12:24:02.543670 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.543598    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:02.545246 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.545222    4464 patch_prober.go:29] interesting pod/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 12:24:02.545442 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.545264    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d" podUID="e834d8c0-89ce-4797-b500-fd61f84ad58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 12:24:02.624421 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.624383    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-kube-api-access-g8lpd\" (UniqueName: \"kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-kube-api-access-g8lpd\") pod \"271e7877-ab08-4f16-9947-5910eb6a29fb\" (UID: \"271e7877-ab08-4f16-9947-5910eb6a29fb\") "
Nov 02 12:24:02.624595 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.624464    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/271e7877-ab08-4f16-9947-5910eb6a29fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271e7877-ab08-4f16-9947-5910eb6a29fb-plugins-config-volume\") pod \"271e7877-ab08-4f16-9947-5910eb6a29fb\" (UID: \"271e7877-ab08-4f16-9947-5910eb6a29fb\") "
Nov 02 12:24:02.624668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.624604    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-provider-credentials\") pod \"271e7877-ab08-4f16-9947-5910eb6a29fb\" (UID: \"271e7877-ab08-4f16-9947-5910eb6a29fb\") "
Nov 02 12:24:02.624726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.624667    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/271e7877-ab08-4f16-9947-5910eb6a29fb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/271e7877-ab08-4f16-9947-5910eb6a29fb-routing-overlay-volume\") pod \"271e7877-ab08-4f16-9947-5910eb6a29fb\" (UID: \"271e7877-ab08-4f16-9947-5910eb6a29fb\") "
Nov 02 12:24:02.625067 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.625035    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271e7877-ab08-4f16-9947-5910eb6a29fb-plugins-config-volume" pod "271e7877-ab08-4f16-9947-5910eb6a29fb" (UID: "271e7877-ab08-4f16-9947-5910eb6a29fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:24:02.625406 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.625383    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/271e7877-ab08-4f16-9947-5910eb6a29fb-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:02.625501 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.625382    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271e7877-ab08-4f16-9947-5910eb6a29fb-routing-overlay-volume" pod "271e7877-ab08-4f16-9947-5910eb6a29fb" (UID: "271e7877-ab08-4f16-9947-5910eb6a29fb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:24:02.627382 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.627358    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-provider-credentials" pod "271e7877-ab08-4f16-9947-5910eb6a29fb" (UID: "271e7877-ab08-4f16-9947-5910eb6a29fb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:24:02.629107 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.629081    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-kube-api-access-g8lpd" pod "271e7877-ab08-4f16-9947-5910eb6a29fb" (UID: "271e7877-ab08-4f16-9947-5910eb6a29fb"). InnerVolumeSpecName "kube-api-access-g8lpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:24:02.726122 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.726091    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8lpd\" (UniqueName: \"kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-kube-api-access-g8lpd\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:02.726122 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.726116    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/271e7877-ab08-4f16-9947-5910eb6a29fb-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:02.726122 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.726126    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/271e7877-ab08-4f16-9947-5910eb6a29fb-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:02.734461 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.734432    4464 generic.go:356] "Generic (PLEG): container finished" podID="271e7877-ab08-4f16-9947-5910eb6a29fb" containerID="d1003463503917ae07da1bef2a1a2c72fa1bbb24df1696e676e9526e7bf5aea9" exitCode=0
Nov 02 12:24:02.734574 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.734480    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"
Nov 02 12:24:02.734574 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.734509    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8" event={"ID":"271e7877-ab08-4f16-9947-5910eb6a29fb","Type":"ContainerDied","Data":"d1003463503917ae07da1bef2a1a2c72fa1bbb24df1696e676e9526e7bf5aea9"}
Nov 02 12:24:02.734574 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.734570    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8" event={"ID":"271e7877-ab08-4f16-9947-5910eb6a29fb","Type":"ContainerDied","Data":"9f6904060a8d8b137a8601ac112a4febf0fc55369650516a9e9840a4f90705d1"}
Nov 02 12:24:02.734725 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.734596    4464 scope.go:122] "RemoveContainer" containerID="d1003463503917ae07da1bef2a1a2c72fa1bbb24df1696e676e9526e7bf5aea9"
Nov 02 12:24:02.745431 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.745413    4464 scope.go:122] "RemoveContainer" containerID="d1003463503917ae07da1bef2a1a2c72fa1bbb24df1696e676e9526e7bf5aea9"
Nov 02 12:24:02.745722 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:02.745702    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1003463503917ae07da1bef2a1a2c72fa1bbb24df1696e676e9526e7bf5aea9\": container with ID starting with d1003463503917ae07da1bef2a1a2c72fa1bbb24df1696e676e9526e7bf5aea9 not found: ID does not exist" containerID="d1003463503917ae07da1bef2a1a2c72fa1bbb24df1696e676e9526e7bf5aea9"
Nov 02 12:24:02.745771 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.745728    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1003463503917ae07da1bef2a1a2c72fa1bbb24df1696e676e9526e7bf5aea9"} err="failed to get container status \"d1003463503917ae07da1bef2a1a2c72fa1bbb24df1696e676e9526e7bf5aea9\": rpc error: code = NotFound desc = could not find container \"d1003463503917ae07da1bef2a1a2c72fa1bbb24df1696e676e9526e7bf5aea9\": container with ID starting with d1003463503917ae07da1bef2a1a2c72fa1bbb24df1696e676e9526e7bf5aea9 not found: ID does not exist"
Nov 02 12:24:02.904810 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.904735    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"]
Nov 02 12:24:02.908600 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:02.908574    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6b87z6xt8"]
Nov 02 12:24:03.543277 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:03.543240    4464 patch_prober.go:29] interesting pod/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 12:24:03.543456 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:03.543302    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d" podUID="e834d8c0-89ce-4797-b500-fd61f84ad58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 12:24:03.592261 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:03.592230    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14b309a8-faca-49c6-93da-9563221f653f" path="/var/lib/kubelet/pods/14b309a8-faca-49c6-93da-9563221f653f/volumes"
Nov 02 12:24:03.592666 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:03.592653    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="271e7877-ab08-4f16-9947-5910eb6a29fb" path="/var/lib/kubelet/pods/271e7877-ab08-4f16-9947-5910eb6a29fb/volumes"
Nov 02 12:24:04.543626 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:04.543594    4464 patch_prober.go:29] interesting pod/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 12:24:04.544040 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:04.543644    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d" podUID="e834d8c0-89ce-4797-b500-fd61f84ad58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 12:24:05.081000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:05.070869    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"]
Nov 02 12:24:05.094597 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:05.091505    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbfad1a-msskc"]
Nov 02 12:24:05.543251 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:05.543217    4464 patch_prober.go:29] interesting pod/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 12:24:05.543434 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:05.543269    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d" podUID="e834d8c0-89ce-4797-b500-fd61f84ad58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 12:24:05.593893 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:05.593861    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4acf9653-c380-411c-a0a3-18f49e9f2c9e" path="/var/lib/kubelet/pods/4acf9653-c380-411c-a0a3-18f49e9f2c9e/volumes"
Nov 02 12:24:06.543518 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:06.543478    4464 patch_prober.go:29] interesting pod/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 12:24:06.543720 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:06.543539    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d" podUID="e834d8c0-89ce-4797-b500-fd61f84ad58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 12:24:06.556357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:06.556326    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"]
Nov 02 12:24:06.612300 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:06.612262    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6a432197-g8dw2"]
Nov 02 12:24:07.048644 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:07.048608    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"]
Nov 02 12:24:07.099105 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:07.099057    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-054acce8-lwsn8"]
Nov 02 12:24:07.543868 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:07.543832    4464 patch_prober.go:29] interesting pod/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 02 12:24:07.544204 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:07.543902    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d" podUID="e834d8c0-89ce-4797-b500-fd61f84ad58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 02 12:24:07.583646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:07.583602    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3" path="/var/lib/kubelet/pods/24ecc964-fea9-4220-bdf0-c2e5d4fa6cc3/volumes"
Nov 02 12:24:07.584174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:07.584156    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feb5d296-405a-4255-812b-8b6f1005b0c3" path="/var/lib/kubelet/pods/feb5d296-405a-4255-812b-8b6f1005b0c3/volumes"
Nov 02 12:24:08.548069 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:08.548042    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:08.548547 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:08.548406    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:08.549119 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:08.549098    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:24:10.161758 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:10.161720    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:24:12.295906 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:12.295858    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx" podUID="d70ff9e4-f0f1-4341-b4f6-dc67961c2255" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:24:12.645609 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:12.645534    4464 scope.go:122] "RemoveContainer" containerID="92607e1cdba75aa52826edfebee09b63f5bd28f2fdc01bacaa9bfaade2e734f7"
Nov 02 12:24:12.656352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:12.656324    4464 scope.go:122] "RemoveContainer" containerID="a43f1833c93358568792d2faa5092208470f43afebb04357c26672466b470b03"
Nov 02 12:24:12.665442 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:12.665420    4464 scope.go:122] "RemoveContainer" containerID="2b75253483ba9bf714c57c4d6bcdfb319ddae7d739bb20db4fa95223582d2882"
Nov 02 12:24:12.674239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:12.674218    4464 scope.go:122] "RemoveContainer" containerID="808490c5d1a889b13ac34c8031a48d67aa924bd663ca98dd461721c63d1badb6"
Nov 02 12:24:12.682706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:12.682684    4464 scope.go:122] "RemoveContainer" containerID="95c37f04c05d7c49258e541108dc055e189645edcfb7be65f2e9118b5804c4fc"
Nov 02 12:24:12.692434 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:12.692417    4464 scope.go:122] "RemoveContainer" containerID="1d9a61cc0e99188e4f1e36a0df29182ef30bf1331cfa3bc50b6b5e27dbb24952"
Nov 02 12:24:14.032940 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:14.032905    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"]
Nov 02 12:24:14.034242 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:14.034212    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx" podUID="d70ff9e4-f0f1-4341-b4f6-dc67961c2255" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:24:19.033466 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:19.033407    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx" podUID="d70ff9e4-f0f1-4341-b4f6-dc67961c2255" containerName="payload-processing" containerID="cri-o://c5e10721fcc1714fc4c8acad4361fe5161bc349ccc20631b50e44a4e1db8ee02" gracePeriod=25
Nov 02 12:24:19.303232 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:19.303208    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"
Nov 02 12:24:19.443298 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:19.443268    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-provider-credentials\") pod \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\" (UID: \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\") "
Nov 02 12:24:19.443500 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:19.443353    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-kube-api-access-rpdv7\" (UniqueName: \"kubernetes.io/projected/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-kube-api-access-rpdv7\") pod \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\" (UID: \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\") "
Nov 02 12:24:19.443500 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:19.443464    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-plugins-config-volume\") pod \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\" (UID: \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\") "
Nov 02 12:24:19.443633 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:19.443497    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-routing-overlay-volume\") pod \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\" (UID: \"d70ff9e4-f0f1-4341-b4f6-dc67961c2255\") "
Nov 02 12:24:19.444225 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:19.443896    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-routing-overlay-volume" pod "d70ff9e4-f0f1-4341-b4f6-dc67961c2255" (UID: "d70ff9e4-f0f1-4341-b4f6-dc67961c2255"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:24:19.444225 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:19.443905    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-plugins-config-volume" pod "d70ff9e4-f0f1-4341-b4f6-dc67961c2255" (UID: "d70ff9e4-f0f1-4341-b4f6-dc67961c2255"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:24:19.444225 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:19.444179    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:19.444225 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:19.444201    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:19.445649 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:19.445619    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-provider-credentials" pod "d70ff9e4-f0f1-4341-b4f6-dc67961c2255" (UID: "d70ff9e4-f0f1-4341-b4f6-dc67961c2255"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:24:19.445926 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:19.445907    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-kube-api-access-rpdv7" pod "d70ff9e4-f0f1-4341-b4f6-dc67961c2255" (UID: "d70ff9e4-f0f1-4341-b4f6-dc67961c2255"). InnerVolumeSpecName "kube-api-access-rpdv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:24:19.545243 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:19.545162    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:19.545243 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:19.545188    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpdv7\" (UniqueName: \"kubernetes.io/projected/d70ff9e4-f0f1-4341-b4f6-dc67961c2255-kube-api-access-rpdv7\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:19.646735 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:19.646703    4464 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-15ea039108c9: secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:19.646735 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:19.646738    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-866d944846-nnbj2: secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:19.646962 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:19.646807    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials podName:8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac nodeName:}" failed. No retries permitted until 2026-11-02 12:24:20.146786782 +0000 UTC m=+1688.710495168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials") pod "payload-processing-external-model-866d944846-nnbj2" (UID: "8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac") : secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:20.049626 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:20.049594    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"]
Nov 02 12:24:20.088841 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:20.088805    4464 generic.go:356] "Generic (PLEG): container finished" podID="d70ff9e4-f0f1-4341-b4f6-dc67961c2255" containerID="c5e10721fcc1714fc4c8acad4361fe5161bc349ccc20631b50e44a4e1db8ee02" exitCode=0
Nov 02 12:24:20.089043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:20.088869    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"
Nov 02 12:24:20.089043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:20.088898    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx" event={"ID":"d70ff9e4-f0f1-4341-b4f6-dc67961c2255","Type":"ContainerDied","Data":"c5e10721fcc1714fc4c8acad4361fe5161bc349ccc20631b50e44a4e1db8ee02"}
Nov 02 12:24:20.089043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:20.089004    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx" event={"ID":"d70ff9e4-f0f1-4341-b4f6-dc67961c2255","Type":"ContainerDied","Data":"197af3768df794de238e57b01357465506adf1b8723beb8de47107de0276b6e0"}
Nov 02 12:24:20.089043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:20.089038    4464 scope.go:122] "RemoveContainer" containerID="c5e10721fcc1714fc4c8acad4361fe5161bc349ccc20631b50e44a4e1db8ee02"
Nov 02 12:24:20.099990 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:20.099952    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9c1ba7-nlpgb"]
Nov 02 12:24:20.100511 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:20.100491    4464 scope.go:122] "RemoveContainer" containerID="c5e10721fcc1714fc4c8acad4361fe5161bc349ccc20631b50e44a4e1db8ee02"
Nov 02 12:24:20.100822 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:20.100774    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e10721fcc1714fc4c8acad4361fe5161bc349ccc20631b50e44a4e1db8ee02\": container with ID starting with c5e10721fcc1714fc4c8acad4361fe5161bc349ccc20631b50e44a4e1db8ee02 not found: ID does not exist" containerID="c5e10721fcc1714fc4c8acad4361fe5161bc349ccc20631b50e44a4e1db8ee02"
Nov 02 12:24:20.100928 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:20.100829    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e10721fcc1714fc4c8acad4361fe5161bc349ccc20631b50e44a4e1db8ee02"} err="failed to get container status \"c5e10721fcc1714fc4c8acad4361fe5161bc349ccc20631b50e44a4e1db8ee02\": rpc error: code = NotFound desc = could not find container \"c5e10721fcc1714fc4c8acad4361fe5161bc349ccc20631b50e44a4e1db8ee02\": container with ID starting with c5e10721fcc1714fc4c8acad4361fe5161bc349ccc20631b50e44a4e1db8ee02 not found: ID does not exist"
Nov 02 12:24:20.151231 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:20.151206    4464 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-15ea039108c9: secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:20.151231 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:20.151231    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-866d944846-nnbj2: secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:20.151396 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:20.151287    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials podName:8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac nodeName:}" failed. No retries permitted until 2026-11-02 12:24:21.151272853 +0000 UTC m=+1689.714981240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials") pod "payload-processing-external-model-866d944846-nnbj2" (UID: "8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac") : secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:20.260781 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:20.260747    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"]
Nov 02 12:24:20.264375 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:20.264343    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-749cdcd88d-dm2dx"]
Nov 02 12:24:21.163761 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:21.163728    4464 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-15ea039108c9: secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:21.163761 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:21.163757    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-866d944846-nnbj2: secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:21.164251 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:21.163814    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials podName:8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac nodeName:}" failed. No retries permitted until 2026-11-02 12:24:23.163799827 +0000 UTC m=+1691.727508214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials") pod "payload-processing-external-model-866d944846-nnbj2" (UID: "8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac") : secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:21.583147 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:21.583116    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d70ff9e4-f0f1-4341-b4f6-dc67961c2255" path="/var/lib/kubelet/pods/d70ff9e4-f0f1-4341-b4f6-dc67961c2255/volumes"
Nov 02 12:24:21.583622 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:21.583606    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec5fbab1-5776-4d45-8beb-19c36dd6467f" path="/var/lib/kubelet/pods/ec5fbab1-5776-4d45-8beb-19c36dd6467f/volumes"
Nov 02 12:24:22.318838 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.318806    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"]
Nov 02 12:24:22.319552 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.319528    4464 cpu_manager.go:414] "containerMap: removing container" podUID="271e7877-ab08-4f16-9947-5910eb6a29fb" containerName="payload-processing"
Nov 02 12:24:22.319552 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.319548    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271e7877-ab08-4f16-9947-5910eb6a29fb" containerName="payload-processing"
Nov 02 12:24:22.319752 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.319573    4464 cpu_manager.go:414] "containerMap: removing container" podUID="14b309a8-faca-49c6-93da-9563221f653f" containerName="payload-processing"
Nov 02 12:24:22.319752 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.319581    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b309a8-faca-49c6-93da-9563221f653f" containerName="payload-processing"
Nov 02 12:24:22.319752 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.319640    4464 cpu_manager.go:414] "containerMap: removing container" podUID="d70ff9e4-f0f1-4341-b4f6-dc67961c2255" containerName="payload-processing"
Nov 02 12:24:22.319752 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.319647    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70ff9e4-f0f1-4341-b4f6-dc67961c2255" containerName="payload-processing"
Nov 02 12:24:22.319752 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.319752    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70ff9e4-f0f1-4341-b4f6-dc67961c2255" containerName="payload-processing"
Nov 02 12:24:22.319939 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.319763    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="271e7877-ab08-4f16-9947-5910eb6a29fb" containerName="payload-processing"
Nov 02 12:24:22.319939 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.319772    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b309a8-faca-49c6-93da-9563221f653f" containerName="payload-processing"
Nov 02 12:24:22.325038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.325019    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"
Nov 02 12:24:22.378310 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.378280    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6tnk\" (UniqueName: \"kubernetes.io/projected/57ff1019-3226-4cdc-adf8-65d3ac66ab70-kube-api-access-v6tnk\") pod \"payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt\" (UID: \"57ff1019-3226-4cdc-adf8-65d3ac66ab70\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"
Nov 02 12:24:22.378545 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.378523    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ff1019-3226-4cdc-adf8-65d3ac66ab70-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt\" (UID: \"57ff1019-3226-4cdc-adf8-65d3ac66ab70\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"
Nov 02 12:24:22.380026 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.380004    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"]
Nov 02 12:24:22.444600 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.432551    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6a432197\""
Nov 02 12:24:22.467503 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.457194    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6a432197-dockercfg-c22dp\""
Nov 02 12:24:22.473902 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.473870    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"]
Nov 02 12:24:22.497319 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.479872    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ff1019-3226-4cdc-adf8-65d3ac66ab70-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt\" (UID: \"57ff1019-3226-4cdc-adf8-65d3ac66ab70\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"
Nov 02 12:24:22.503006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.480922    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ff1019-3226-4cdc-adf8-65d3ac66ab70-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt\" (UID: \"57ff1019-3226-4cdc-adf8-65d3ac66ab70\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"
Nov 02 12:24:22.503256 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.503222    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6tnk\" (UniqueName: \"kubernetes.io/projected/57ff1019-3226-4cdc-adf8-65d3ac66ab70-kube-api-access-v6tnk\") pod \"payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt\" (UID: \"57ff1019-3226-4cdc-adf8-65d3ac66ab70\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"
Nov 02 12:24:22.569093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.561959    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"
Nov 02 12:24:22.569093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.566967    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"]
Nov 02 12:24:22.571840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.570938    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6tnk\" (UniqueName: \"kubernetes.io/projected/57ff1019-3226-4cdc-adf8-65d3ac66ab70-kube-api-access-v6tnk\") pod \"payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt\" (UID: \"57ff1019-3226-4cdc-adf8-65d3ac66ab70\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"
Nov 02 12:24:22.651254 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.651199    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"]
Nov 02 12:24:22.662661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.652264    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"
Nov 02 12:24:22.662661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.656183    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:24:22.662661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.657747    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:24:22.662661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.658183    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:24:22.669176 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.666074    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:24:22.669992 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.669463    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glkpl\" (UniqueName: \"kubernetes.io/projected/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-kube-api-access-glkpl\") pod \"maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8\" (UID: \"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"
Nov 02 12:24:22.669992 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.669714    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8\" (UID: \"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"
Nov 02 12:24:22.670180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.670003    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8\" (UID: \"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"
Nov 02 12:24:22.698024 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.697773    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"]
Nov 02 12:24:22.698024 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.697869    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"
Nov 02 12:24:22.853084 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.792380    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8\" (UID: \"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"
Nov 02 12:24:22.853084 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.792484    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k\" (UID: \"e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f\") " pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"
Nov 02 12:24:22.853084 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.792540    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glkpl\" (UniqueName: \"kubernetes.io/projected/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-kube-api-access-glkpl\") pod \"maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8\" (UID: \"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"
Nov 02 12:24:22.853084 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.792597    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s7d2\" (UniqueName: \"kubernetes.io/projected/e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f-kube-api-access-5s7d2\") pod \"payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k\" (UID: \"e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f\") " pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"
Nov 02 12:24:22.853084 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.792803    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8\" (UID: \"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"
Nov 02 12:24:22.853084 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.794002    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8\" (UID: \"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"
Nov 02 12:24:22.907835 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.906071    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k\" (UID: \"e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f\") " pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"
Nov 02 12:24:22.907835 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.906246    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s7d2\" (UniqueName: \"kubernetes.io/projected/e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f-kube-api-access-5s7d2\") pod \"payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k\" (UID: \"e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f\") " pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"
Nov 02 12:24:22.907835 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.907606    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f-plugins-config-volume\") pod \"payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k\" (UID: \"e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f\") " pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"
Nov 02 12:24:22.949094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.949055    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8\" (UID: \"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"
Nov 02 12:24:22.959293 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.959256    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glkpl\" (UniqueName: \"kubernetes.io/projected/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-kube-api-access-glkpl\") pod \"maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8\" (UID: \"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"
Nov 02 12:24:22.969784 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.969755    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s7d2\" (UniqueName: \"kubernetes.io/projected/e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f-kube-api-access-5s7d2\") pod \"payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k\" (UID: \"e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f\") " pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"
Nov 02 12:24:22.985432 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:22.985402    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"
Nov 02 12:24:23.033281 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:23.033244    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"
Nov 02 12:24:23.054194 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:23.054153    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"]
Nov 02 12:24:23.059282 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:23.059240    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d755c31c-4jg9p"]
Nov 02 12:24:23.210619 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:23.210586    4464 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-15ea039108c9: secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:23.210619 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:23.210615    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-866d944846-nnbj2: secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:23.210803 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:23.210671    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials podName:8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac nodeName:}" failed. No retries permitted until 2026-11-02 12:24:27.21065529 +0000 UTC m=+1695.774363681 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials") pod "payload-processing-external-model-866d944846-nnbj2" (UID: "8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac") : secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:23.274448 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:23.274420    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"]
Nov 02 12:24:23.274762 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:24:23.274741    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ff1019_3226_4cdc_adf8_65d3ac66ab70.slice/crio-299636b10b9aca888c13b265afdad26344c0a9a882c97d8ecee76a71dc62f266 WatchSource:0}: Error finding container 299636b10b9aca888c13b265afdad26344c0a9a882c97d8ecee76a71dc62f266: Status 404 returned error can't find the container with id 299636b10b9aca888c13b265afdad26344c0a9a882c97d8ecee76a71dc62f266
Nov 02 12:24:23.371295 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:24:23.371266    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2e0964_1eaf_4f6a_92dc_fbfb2ba31df5.slice/crio-f6ca6493bfb7c7f47e3ba98fce5832406cca27f17fc2b0fd96ebe18eace3fb7c WatchSource:0}: Error finding container f6ca6493bfb7c7f47e3ba98fce5832406cca27f17fc2b0fd96ebe18eace3fb7c: Status 404 returned error can't find the container with id f6ca6493bfb7c7f47e3ba98fce5832406cca27f17fc2b0fd96ebe18eace3fb7c
Nov 02 12:24:23.372222 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:23.372198    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"]
Nov 02 12:24:23.425275 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:24:23.425235    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23de9bc_9894_4a21_8e6f_c60b3b9c8e9f.slice/crio-4c5b5571fdcd530e6420af9b3d2b2dcac6bd213b4c5b1bf03a13f42c61cb64ca WatchSource:0}: Error finding container 4c5b5571fdcd530e6420af9b3d2b2dcac6bd213b4c5b1bf03a13f42c61cb64ca: Status 404 returned error can't find the container with id 4c5b5571fdcd530e6420af9b3d2b2dcac6bd213b4c5b1bf03a13f42c61cb64ca
Nov 02 12:24:23.425395 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:23.425375    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"]
Nov 02 12:24:23.584871 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:23.584837    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3af88032-0c31-4bb0-a7a9-032f74417e69" path="/var/lib/kubelet/pods/3af88032-0c31-4bb0-a7a9-032f74417e69/volumes"
Nov 02 12:24:24.123169 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.123131    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k" event={"ID":"e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f","Type":"ContainerStarted","Data":"0074432e965b2065248ad025df88b1f82fb31b45e02966a04f3bd8786d74d027"}
Nov 02 12:24:24.123374 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.123188    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k" event={"ID":"e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f","Type":"ContainerStarted","Data":"4c5b5571fdcd530e6420af9b3d2b2dcac6bd213b4c5b1bf03a13f42c61cb64ca"}
Nov 02 12:24:24.123374 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.123216    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"
Nov 02 12:24:24.124969 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.124944    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8" event={"ID":"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5","Type":"ContainerStarted","Data":"52a661decff4615686f8a4c9fec8b55bbf08932afdec776da427d94696bac20f"}
Nov 02 12:24:24.125111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.125007    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8" event={"ID":"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5","Type":"ContainerStarted","Data":"f6ca6493bfb7c7f47e3ba98fce5832406cca27f17fc2b0fd96ebe18eace3fb7c"}
Nov 02 12:24:24.126619 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.126598    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt" event={"ID":"57ff1019-3226-4cdc-adf8-65d3ac66ab70","Type":"ContainerStarted","Data":"4eb9275c988516617c2ce30708c8efa9eb4856b2233ff3fdfde004cf1e9d5c74"}
Nov 02 12:24:24.126686 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.126636    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt" event={"ID":"57ff1019-3226-4cdc-adf8-65d3ac66ab70","Type":"ContainerStarted","Data":"299636b10b9aca888c13b265afdad26344c0a9a882c97d8ecee76a71dc62f266"}
Nov 02 12:24:24.126686 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.126667    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"
Nov 02 12:24:24.245345 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.245297    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k" podStartSLOduration=2.245283489 podStartE2EDuration="2.245283489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:24:24.241655492 +0000 UTC m=+1692.805363902" watchObservedRunningTime="2026-11-02 12:24:24.245283489 +0000 UTC m=+1692.808991897"
Nov 02 12:24:24.324704 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.324675    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"]
Nov 02 12:24:24.329080 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.329060    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:24:24.361828 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.361784    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8" podStartSLOduration=2.361771471 podStartE2EDuration="2.361771471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:24:24.360173339 +0000 UTC m=+1692.923881752" watchObservedRunningTime="2026-11-02 12:24:24.361771471 +0000 UTC m=+1692.925479879"
Nov 02 12:24:24.382547 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.382466    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bef5ec4d23f0\""
Nov 02 12:24:24.382547 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.382484    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1c6bea6087ce\""
Nov 02 12:24:24.386607 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.386582    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"]
Nov 02 12:24:24.462997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.453713    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbzn8\" (UniqueName: \"kubernetes.io/projected/1068cf8a-7fc4-48ea-9522-3fc6e914891a-kube-api-access-xbzn8\") pod \"payload-processing-external-model-5cc9844f6-pv7vm\" (UID: \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\") " pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:24:24.462997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.453816    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1068cf8a-7fc4-48ea-9522-3fc6e914891a-routing-overlay-volume\") pod \"payload-processing-external-model-5cc9844f6-pv7vm\" (UID: \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\") " pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:24:24.462997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.453960    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1068cf8a-7fc4-48ea-9522-3fc6e914891a-provider-credentials\") pod \"payload-processing-external-model-5cc9844f6-pv7vm\" (UID: \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\") " pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:24:24.462997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.454059    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1068cf8a-7fc4-48ea-9522-3fc6e914891a-plugins-config-volume\") pod \"payload-processing-external-model-5cc9844f6-pv7vm\" (UID: \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\") " pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:24:24.555617 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.555576    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1068cf8a-7fc4-48ea-9522-3fc6e914891a-plugins-config-volume\") pod \"payload-processing-external-model-5cc9844f6-pv7vm\" (UID: \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\") " pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:24:24.555814 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.555685    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbzn8\" (UniqueName: \"kubernetes.io/projected/1068cf8a-7fc4-48ea-9522-3fc6e914891a-kube-api-access-xbzn8\") pod \"payload-processing-external-model-5cc9844f6-pv7vm\" (UID: \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\") " pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:24:24.555814 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.555726    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1068cf8a-7fc4-48ea-9522-3fc6e914891a-routing-overlay-volume\") pod \"payload-processing-external-model-5cc9844f6-pv7vm\" (UID: \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\") " pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:24:24.555942 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.555819    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1068cf8a-7fc4-48ea-9522-3fc6e914891a-provider-credentials\") pod \"payload-processing-external-model-5cc9844f6-pv7vm\" (UID: \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\") " pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:24:24.556365 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.556340    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1068cf8a-7fc4-48ea-9522-3fc6e914891a-routing-overlay-volume\") pod \"payload-processing-external-model-5cc9844f6-pv7vm\" (UID: \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\") " pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:24:24.556508 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.556484    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1068cf8a-7fc4-48ea-9522-3fc6e914891a-plugins-config-volume\") pod \"payload-processing-external-model-5cc9844f6-pv7vm\" (UID: \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\") " pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:24:24.558318 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.558297    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1068cf8a-7fc4-48ea-9522-3fc6e914891a-provider-credentials\") pod \"payload-processing-external-model-5cc9844f6-pv7vm\" (UID: \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\") " pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:24:24.570207 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.570168    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt" podStartSLOduration=2.5701556979999998 podStartE2EDuration="2.570155698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:24:24.568596737 +0000 UTC m=+1693.132305165" watchObservedRunningTime="2026-11-02 12:24:24.570155698 +0000 UTC m=+1693.133864106"
Nov 02 12:24:24.617746 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.617716    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbzn8\" (UniqueName: \"kubernetes.io/projected/1068cf8a-7fc4-48ea-9522-3fc6e914891a-kube-api-access-xbzn8\") pod \"payload-processing-external-model-5cc9844f6-pv7vm\" (UID: \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\") " pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:24:24.654165 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:24.654106    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:24:25.102663 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:24:25.102630    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1068cf8a_7fc4_48ea_9522_3fc6e914891a.slice/crio-fecd13ded6c399cf705f6230e6aa7ec4f0d1cade96bd27db681ec791ed7f1ef2 WatchSource:0}: Error finding container fecd13ded6c399cf705f6230e6aa7ec4f0d1cade96bd27db681ec791ed7f1ef2: Status 404 returned error can't find the container with id fecd13ded6c399cf705f6230e6aa7ec4f0d1cade96bd27db681ec791ed7f1ef2
Nov 02 12:24:25.104361 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:25.104335    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"]
Nov 02 12:24:25.135621 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:25.135592    4464 generic.go:356] "Generic (PLEG): container finished" podID="2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5" containerID="52a661decff4615686f8a4c9fec8b55bbf08932afdec776da427d94696bac20f" exitCode=0
Nov 02 12:24:25.135744 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:25.135667    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8" event={"ID":"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5","Type":"ContainerDied","Data":"52a661decff4615686f8a4c9fec8b55bbf08932afdec776da427d94696bac20f"}
Nov 02 12:24:25.136957 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:25.136935    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm" event={"ID":"1068cf8a-7fc4-48ea-9522-3fc6e914891a","Type":"ContainerStarted","Data":"fecd13ded6c399cf705f6230e6aa7ec4f0d1cade96bd27db681ec791ed7f1ef2"}
Nov 02 12:24:26.147730 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:26.147681    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm" event={"ID":"1068cf8a-7fc4-48ea-9522-3fc6e914891a","Type":"ContainerStarted","Data":"b312bf4d7aa60ff6f73723af08c3aa7c1dd28cd97e3385fe01d233e159d6a1cb"}
Nov 02 12:24:26.148234 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:26.147951    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:24:26.267164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:26.267112    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm" podStartSLOduration=2.267098079 podStartE2EDuration="2.267098079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:24:26.26633752 +0000 UTC m=+1694.830045941" watchObservedRunningTime="2026-11-02 12:24:26.267098079 +0000 UTC m=+1694.830806488"
Nov 02 12:24:26.307484 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:26.307458    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"
Nov 02 12:24:26.382947 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:26.382919    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-kube-api-access-glkpl\" (UniqueName: \"kubernetes.io/projected/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-kube-api-access-glkpl\") pod \"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5\" (UID: \"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5\") "
Nov 02 12:24:26.383197 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:26.382965    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-maas-api-xks-ca\") pod \"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5\" (UID: \"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5\") "
Nov 02 12:24:26.383197 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:26.383189    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-maas-api-openshift-ca\") pod \"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5\" (UID: \"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5\") "
Nov 02 12:24:26.383568 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:26.383529    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-maas-api-openshift-ca" pod "2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5" (UID: "2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:24:26.383700 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:26.383681    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:26.385185 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:26.385165    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-maas-api-xks-ca" pod "2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5" (UID: "2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:24:26.385256 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:26.385228    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-kube-api-access-glkpl" pod "2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5" (UID: "2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5"). InnerVolumeSpecName "kube-api-access-glkpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:24:26.485063 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:26.485024    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glkpl\" (UniqueName: \"kubernetes.io/projected/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-kube-api-access-glkpl\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:26.485063 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:26.485062    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:27.155778 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:27.155743    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"
Nov 02 12:24:27.156186 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:27.155775    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8" event={"ID":"2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5","Type":"ContainerDied","Data":"f6ca6493bfb7c7f47e3ba98fce5832406cca27f17fc2b0fd96ebe18eace3fb7c"}
Nov 02 12:24:27.156186 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:27.155824    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6ca6493bfb7c7f47e3ba98fce5832406cca27f17fc2b0fd96ebe18eace3fb7c"
Nov 02 12:24:27.295236 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:27.295202    4464 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-15ea039108c9: secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:27.295236 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:27.295237    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-866d944846-nnbj2: secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:27.295476 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:27.295314    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials podName:8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac nodeName:}" failed. No retries permitted until 2026-11-02 12:24:35.295297002 +0000 UTC m=+1703.859005389 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials") pod "payload-processing-external-model-866d944846-nnbj2" (UID: "8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac") : secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:28.293853 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:28.293814    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:24:29.807338 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:29.807301    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:24:33.066000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:33.065567    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"]
Nov 02 12:24:33.134999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:33.134243    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b641c6c5-vbnw4"]
Nov 02 12:24:33.785702 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:33.785659    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f14d4e2-8f92-4a32-885e-5a132b131e72" path="/var/lib/kubelet/pods/6f14d4e2-8f92-4a32-885e-5a132b131e72/volumes"
Nov 02 12:24:35.004999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:35.004120    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"]
Nov 02 12:24:35.146017 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:35.145067    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"]
Nov 02 12:24:35.164835 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:35.164126    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"
Nov 02 12:24:35.177144 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:35.177116    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"
Nov 02 12:24:35.380991 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:35.364238    4464 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-15ea039108c9: secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:35.387208 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:35.387179    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-866d944846-nnbj2: secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:35.400020 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:35.399994    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials podName:8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac nodeName:}" failed. No retries permitted until 2026-11-02 12:24:51.399945758 +0000 UTC m=+1719.963654168 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials") pod "payload-processing-external-model-866d944846-nnbj2" (UID: "8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac") : secret "agc-cred-15ea039108c9" not found
Nov 02 12:24:35.911078 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:35.911047    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"]
Nov 02 12:24:35.911684 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:35.911671    4464 cpu_manager.go:414] "containerMap: removing container" podUID="2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5" containerName="revoke-keys"
Nov 02 12:24:35.911732 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:35.911685    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5" containerName="revoke-keys"
Nov 02 12:24:35.911805 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:35.911796    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5" containerName="revoke-keys"
Nov 02 12:24:35.915421 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:35.915401    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:35.977396 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:35.977366    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"]
Nov 02 12:24:36.000800 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.000765    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-workload-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.001020 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.000805    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istiod-ca-cert\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.001020 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.000844    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-podinfo\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.001020 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.000875    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-envoy\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.001020 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.000912    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-workload-certs\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.001020 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.000953    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-credential-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.001340 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.001027    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-token\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.001340 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.001061    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crscp\" (UniqueName: \"kubernetes.io/projected/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-kube-api-access-crscp\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.001340 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.001092    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-data\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.102395 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.102364    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-workload-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.102395 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.102398    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istiod-ca-cert\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.102907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.102427    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-podinfo\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.102907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.102451    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-envoy\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.102907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.102486    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-workload-certs\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.102907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.102526    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-credential-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.102907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.102582    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-token\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.102907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.102604    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crscp\" (UniqueName: \"kubernetes.io/projected/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-kube-api-access-crscp\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.102907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.102628    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-data\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.102907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.102824    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-workload-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.103360 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.103278    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istiod-ca-cert\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.103360 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.103283    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-data\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.103360 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.103332    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-workload-certs\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.103360 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.103358    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-credential-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.104825 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.104800    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-envoy\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.105067 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.105051    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-podinfo\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.164479 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.164428    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-token\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.164716 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.164694    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crscp\" (UniqueName: \"kubernetes.io/projected/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-kube-api-access-crscp\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.229689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.229661    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:36.669322 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.669287    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"]
Nov 02 12:24:36.674368 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.674325    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:24:36.674456 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.674437    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:24:36.674519 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:36.674483    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:24:37.159008 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:37.158962    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:24:37.332008 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:37.331962    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d944846-nnbj2"]
Nov 02 12:24:37.433804 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:37.433716    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp" event={"ID":"09dda7c2-acb9-43ef-a3c2-e7147de8d3be","Type":"ContainerStarted","Data":"ac0bf84e548d8b4dac8619246abb0c9b741a77d84c9c4b394d9eae450217a967"}
Nov 02 12:24:37.433804 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:37.433778    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp" event={"ID":"09dda7c2-acb9-43ef-a3c2-e7147de8d3be","Type":"ContainerStarted","Data":"9e65077e9bac53fe81846cb6df8012eafed7df7a60d55b5070648edd58a33a7d"}
Nov 02 12:24:37.592677 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:37.592617    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp" podStartSLOduration=2.592600712 podStartE2EDuration="2.592600712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:24:37.588270567 +0000 UTC m=+1706.151978979" watchObservedRunningTime="2026-11-02 12:24:37.592600712 +0000 UTC m=+1706.156309118"
Nov 02 12:24:38.230725 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:38.230694    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:38.232452 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:38.232422    4464 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 02 12:24:38.232587 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:38.232471    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp" podUID="09dda7c2-acb9-43ef-a3c2-e7147de8d3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 02 12:24:39.230291 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:39.230260    4464 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 02 12:24:39.230546 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:39.230313    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp" podUID="09dda7c2-acb9-43ef-a3c2-e7147de8d3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 02 12:24:40.005357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.005301    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g" podUID="bfc9abd9-14d6-462a-90f8-a486d9270087" containerName="payload-processing" containerID="cri-o://646fbbff6487e4624066cee9a8f0ca2447d1fa01026c7bfbc7ed4edb859824ba" gracePeriod=25
Nov 02 12:24:40.105765 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:24:40.105735    4464 logging.go:55] [core] [Channel #3479 SubChannel #3481]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.149:50052", ServerName: "10.133.0.149:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.149:50052: connect: connection refused"
Nov 02 12:24:40.146340 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.146304    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq" podUID="4e7814e6-b986-4c57-8c69-91310ce6e283" containerName="payload-pre-processing" containerID="cri-o://9c3c46e8080118aff1e36c5af62128e9386474f6a40440bd3f7c382e9739f72f" gracePeriod=25
Nov 02 12:24:40.231244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.231208    4464 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 02 12:24:40.231389 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.231273    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp" podUID="09dda7c2-acb9-43ef-a3c2-e7147de8d3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 02 12:24:40.328311 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.328282    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"
Nov 02 12:24:40.494274 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.493005    4464 generic.go:356] "Generic (PLEG): container finished" podID="4e7814e6-b986-4c57-8c69-91310ce6e283" containerID="9c3c46e8080118aff1e36c5af62128e9386474f6a40440bd3f7c382e9739f72f" exitCode=0
Nov 02 12:24:40.494274 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.493158    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq" event={"ID":"4e7814e6-b986-4c57-8c69-91310ce6e283","Type":"ContainerDied","Data":"9c3c46e8080118aff1e36c5af62128e9386474f6a40440bd3f7c382e9739f72f"}
Nov 02 12:24:40.494274 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.493207    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq" event={"ID":"4e7814e6-b986-4c57-8c69-91310ce6e283","Type":"ContainerDied","Data":"1b2a6d447d44616367a65e4addd288626379cbcb77dc51992462dcfffc911f67"}
Nov 02 12:24:40.494274 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.493231    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b2a6d447d44616367a65e4addd288626379cbcb77dc51992462dcfffc911f67"
Nov 02 12:24:40.495197 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.495162    4464 generic.go:356] "Generic (PLEG): container finished" podID="bfc9abd9-14d6-462a-90f8-a486d9270087" containerID="646fbbff6487e4624066cee9a8f0ca2447d1fa01026c7bfbc7ed4edb859824ba" exitCode=0
Nov 02 12:24:40.495480 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.495386    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g" event={"ID":"bfc9abd9-14d6-462a-90f8-a486d9270087","Type":"ContainerDied","Data":"646fbbff6487e4624066cee9a8f0ca2447d1fa01026c7bfbc7ed4edb859824ba"}
Nov 02 12:24:40.495480 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.495433    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g" event={"ID":"bfc9abd9-14d6-462a-90f8-a486d9270087","Type":"ContainerDied","Data":"d2a2d48af95b801f51a422cafd747a2a2e3f2672f4a5b8c803b51281a62b2cea"}
Nov 02 12:24:40.495480 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.495463    4464 scope.go:122] "RemoveContainer" containerID="646fbbff6487e4624066cee9a8f0ca2447d1fa01026c7bfbc7ed4edb859824ba"
Nov 02 12:24:40.497212 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.495625    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"
Nov 02 12:24:40.504541 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.504518    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"
Nov 02 12:24:40.509274 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.509256    4464 scope.go:122] "RemoveContainer" containerID="646fbbff6487e4624066cee9a8f0ca2447d1fa01026c7bfbc7ed4edb859824ba"
Nov 02 12:24:40.509544 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:40.509518    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646fbbff6487e4624066cee9a8f0ca2447d1fa01026c7bfbc7ed4edb859824ba\": container with ID starting with 646fbbff6487e4624066cee9a8f0ca2447d1fa01026c7bfbc7ed4edb859824ba not found: ID does not exist" containerID="646fbbff6487e4624066cee9a8f0ca2447d1fa01026c7bfbc7ed4edb859824ba"
Nov 02 12:24:40.509634 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.509549    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646fbbff6487e4624066cee9a8f0ca2447d1fa01026c7bfbc7ed4edb859824ba"} err="failed to get container status \"646fbbff6487e4624066cee9a8f0ca2447d1fa01026c7bfbc7ed4edb859824ba\": rpc error: code = NotFound desc = could not find container \"646fbbff6487e4624066cee9a8f0ca2447d1fa01026c7bfbc7ed4edb859824ba\": container with ID starting with 646fbbff6487e4624066cee9a8f0ca2447d1fa01026c7bfbc7ed4edb859824ba not found: ID does not exist"
Nov 02 12:24:40.514594 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.514563    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfc9abd9-14d6-462a-90f8-a486d9270087-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc9abd9-14d6-462a-90f8-a486d9270087-plugins-config-volume\") pod \"bfc9abd9-14d6-462a-90f8-a486d9270087\" (UID: \"bfc9abd9-14d6-462a-90f8-a486d9270087\") "
Nov 02 12:24:40.514744 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.514730    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc9abd9-14d6-462a-90f8-a486d9270087-kube-api-access-nm954\" (UniqueName: \"kubernetes.io/projected/bfc9abd9-14d6-462a-90f8-a486d9270087-kube-api-access-nm954\") pod \"bfc9abd9-14d6-462a-90f8-a486d9270087\" (UID: \"bfc9abd9-14d6-462a-90f8-a486d9270087\") "
Nov 02 12:24:40.514933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.514909    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc9abd9-14d6-462a-90f8-a486d9270087-plugins-config-volume" pod "bfc9abd9-14d6-462a-90f8-a486d9270087" (UID: "bfc9abd9-14d6-462a-90f8-a486d9270087"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:24:40.515252 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.515236    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc9abd9-14d6-462a-90f8-a486d9270087-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:40.516908 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.516886    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc9abd9-14d6-462a-90f8-a486d9270087-kube-api-access-nm954" pod "bfc9abd9-14d6-462a-90f8-a486d9270087" (UID: "bfc9abd9-14d6-462a-90f8-a486d9270087"). InnerVolumeSpecName "kube-api-access-nm954". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:24:40.616029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.615934    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7814e6-b986-4c57-8c69-91310ce6e283-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7814e6-b986-4c57-8c69-91310ce6e283-plugins-config-volume\") pod \"4e7814e6-b986-4c57-8c69-91310ce6e283\" (UID: \"4e7814e6-b986-4c57-8c69-91310ce6e283\") "
Nov 02 12:24:40.616183 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.616139    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7814e6-b986-4c57-8c69-91310ce6e283-kube-api-access-xc7w8\" (UniqueName: \"kubernetes.io/projected/4e7814e6-b986-4c57-8c69-91310ce6e283-kube-api-access-xc7w8\") pod \"4e7814e6-b986-4c57-8c69-91310ce6e283\" (UID: \"4e7814e6-b986-4c57-8c69-91310ce6e283\") "
Nov 02 12:24:40.616366 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.616344    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7814e6-b986-4c57-8c69-91310ce6e283-plugins-config-volume" pod "4e7814e6-b986-4c57-8c69-91310ce6e283" (UID: "4e7814e6-b986-4c57-8c69-91310ce6e283"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:24:40.616628 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.616604    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7814e6-b986-4c57-8c69-91310ce6e283-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:40.616628 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.616621    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm954\" (UniqueName: \"kubernetes.io/projected/bfc9abd9-14d6-462a-90f8-a486d9270087-kube-api-access-nm954\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:40.618296 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.618267    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7814e6-b986-4c57-8c69-91310ce6e283-kube-api-access-xc7w8" pod "4e7814e6-b986-4c57-8c69-91310ce6e283" (UID: "4e7814e6-b986-4c57-8c69-91310ce6e283"). InnerVolumeSpecName "kube-api-access-xc7w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:24:40.718216 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.718184    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc7w8\" (UniqueName: \"kubernetes.io/projected/4e7814e6-b986-4c57-8c69-91310ce6e283-kube-api-access-xc7w8\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:40.919404 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.919325    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"]
Nov 02 12:24:40.931786 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:40.931746    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g"]
Nov 02 12:24:41.049625 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.049589    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"]
Nov 02 12:24:41.053146 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.053123    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c9536ab-sgclr"]
Nov 02 12:24:41.106209 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.106172    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-migrate-1cc3a7b3-6f865874bc-t6n8g" podUID="bfc9abd9-14d6-462a-90f8-a486d9270087" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.149:50052\" within 1s: context deadline exceeded"
Nov 02 12:24:41.230812 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.230783    4464 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 02 12:24:41.230987 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.230830    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp" podUID="09dda7c2-acb9-43ef-a3c2-e7147de8d3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 02 12:24:41.322306 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.322274    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"]
Nov 02 12:24:41.322946 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.322929    4464 cpu_manager.go:414] "containerMap: removing container" podUID="4e7814e6-b986-4c57-8c69-91310ce6e283" containerName="payload-pre-processing"
Nov 02 12:24:41.323040 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.322950    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7814e6-b986-4c57-8c69-91310ce6e283" containerName="payload-pre-processing"
Nov 02 12:24:41.323040 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.323006    4464 cpu_manager.go:414] "containerMap: removing container" podUID="bfc9abd9-14d6-462a-90f8-a486d9270087" containerName="payload-processing"
Nov 02 12:24:41.323040 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.323013    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfc9abd9-14d6-462a-90f8-a486d9270087" containerName="payload-processing"
Nov 02 12:24:41.323158 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.323116    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfc9abd9-14d6-462a-90f8-a486d9270087" containerName="payload-processing"
Nov 02 12:24:41.323158 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.323129    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7814e6-b986-4c57-8c69-91310ce6e283" containerName="payload-pre-processing"
Nov 02 12:24:41.328012 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.327968    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.395611 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.395577    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-03c4e8\"/\"openshift-service-ca.crt\""
Nov 02 12:24:41.395792 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.395711    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-03c4e8\"/\"kube-root-ca.crt\""
Nov 02 12:24:41.396377 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.396325    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-03c4e8\"/\"default-dockercfg-7bxqj\""
Nov 02 12:24:41.396749 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.396341    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-03c4e8\"/\"sub-test-model-03c4e8-kserve-self-signed-certs\""
Nov 02 12:24:41.401297 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.401276    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"]
Nov 02 12:24:41.426313 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.426285    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84mbx\" (UniqueName: \"kubernetes.io/projected/8eb017b2-26e3-450a-953f-e348a1109d20-kube-api-access-84mbx\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.426434 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.426325    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-home\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.426434 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.426352    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb017b2-26e3-450a-953f-e348a1109d20-tls-certs\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.426519 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.426449    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-dshm\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.426576 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.426514    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-tmp-dir\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.426686 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.426668    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-model-cache\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.530061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.525665    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"
Nov 02 12:24:41.530061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.527858    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-model-cache\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.530061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.528902    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84mbx\" (UniqueName: \"kubernetes.io/projected/8eb017b2-26e3-450a-953f-e348a1109d20-kube-api-access-84mbx\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.530061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.529252    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-home\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.530061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.529551    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-home\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.530061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.529662    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-model-cache\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.548007 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.529761    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb017b2-26e3-450a-953f-e348a1109d20-tls-certs\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.548007 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.546534    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-dshm\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.548007 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.546700    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-tmp-dir\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.548007 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.547297    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-tmp-dir\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.576499 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.570882    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-dshm\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.613005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.598074    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb017b2-26e3-450a-953f-e348a1109d20-tls-certs\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.635001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.634581    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84mbx\" (UniqueName: \"kubernetes.io/projected/8eb017b2-26e3-450a-953f-e348a1109d20-kube-api-access-84mbx\") pod \"sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.647368 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.646545    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:41.650287 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.649772    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad2bbef1-1336-4164-a083-c15cf501bfef" path="/var/lib/kubelet/pods/ad2bbef1-1336-4164-a083-c15cf501bfef/volumes"
Nov 02 12:24:41.652692 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.650524    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfc9abd9-14d6-462a-90f8-a486d9270087" path="/var/lib/kubelet/pods/bfc9abd9-14d6-462a-90f8-a486d9270087/volumes"
Nov 02 12:24:41.987057 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.982798    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"]
Nov 02 12:24:41.988071 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:41.988044    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-1cc3a7b3-5699648675-dcrpq"]
Nov 02 12:24:42.089423 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.089374    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"]
Nov 02 12:24:42.230815 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.230782    4464 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 02 12:24:42.231004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.230834    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp" podUID="09dda7c2-acb9-43ef-a3c2-e7147de8d3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 02 12:24:42.332346 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.332247    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2" podUID="8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac" containerName="payload-processing" containerID="cri-o://1c843bcd4cc5a173c66216d8892e3db90897feb8d66956a13cba2be0ec2a01c2" gracePeriod=25
Nov 02 12:24:42.551701 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.548012    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm" event={"ID":"8eb017b2-26e3-450a-953f-e348a1109d20","Type":"ContainerStarted","Data":"1f1c2f806321de151e09bff07748ece1107cb55d033d54769ea19a6c3f0e2a12"}
Nov 02 12:24:42.551701 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.548070    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm" event={"ID":"8eb017b2-26e3-450a-953f-e348a1109d20","Type":"ContainerStarted","Data":"6cee3b0e4e47896e92e35b1d7fba0ddfaebeef1edcde2d07de03b270c8971eea"}
Nov 02 12:24:42.551701 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.551338    4464 generic.go:356] "Generic (PLEG): container finished" podID="8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac" containerID="1c843bcd4cc5a173c66216d8892e3db90897feb8d66956a13cba2be0ec2a01c2" exitCode=0
Nov 02 12:24:42.551701 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.551430    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2" event={"ID":"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac","Type":"ContainerDied","Data":"1c843bcd4cc5a173c66216d8892e3db90897feb8d66956a13cba2be0ec2a01c2"}
Nov 02 12:24:42.628933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.628908    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:24:42.649211 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.649188    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:42.689997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.688961    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials\") pod \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\" (UID: \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\") "
Nov 02 12:24:42.689997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.689084    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-plugins-config-volume\") pod \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\" (UID: \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\") "
Nov 02 12:24:42.689997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.689129    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-routing-overlay-volume\") pod \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\" (UID: \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\") "
Nov 02 12:24:42.689997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.689186    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-kube-api-access-75bw4\" (UniqueName: \"kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-kube-api-access-75bw4\") pod \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\" (UID: \"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac\") "
Nov 02 12:24:42.689997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.689617    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-plugins-config-volume" pod "8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac" (UID: "8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:24:42.689997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.689916    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:42.690455 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.690365    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-routing-overlay-volume" pod "8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac" (UID: "8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:24:42.693765 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.693732    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-kube-api-access-75bw4" pod "8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac" (UID: "8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac"). InnerVolumeSpecName "kube-api-access-75bw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:24:42.693888 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.693867    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials" pod "8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac" (UID: "8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:24:42.790919 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.790881    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:42.790919 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.790915    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:42.791140 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.790929    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75bw4\" (UniqueName: \"kubernetes.io/projected/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac-kube-api-access-75bw4\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:24:42.812864 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:42.812789    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm" podStartSLOduration=1.812770497 podStartE2EDuration="1.812770497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:24:42.690834127 +0000 UTC m=+1711.254542539" watchObservedRunningTime="2026-11-02 12:24:42.812770497 +0000 UTC m=+1711.376478907"
Nov 02 12:24:43.230440 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:43.230406    4464 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 02 12:24:43.230830 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:43.230466    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp" podUID="09dda7c2-acb9-43ef-a3c2-e7147de8d3be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 02 12:24:43.560205 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:43.560167    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2"
Nov 02 12:24:43.560205 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:43.560189    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-866d944846-nnbj2" event={"ID":"8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac","Type":"ContainerDied","Data":"91ceeae6d91d500e7337d440f84d31ff4255b7ebcae27b6c65613db74cc729a9"}
Nov 02 12:24:43.560468 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:43.560256    4464 scope.go:122] "RemoveContainer" containerID="1c843bcd4cc5a173c66216d8892e3db90897feb8d66956a13cba2be0ec2a01c2"
Nov 02 12:24:43.582526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:43.582502    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e7814e6-b986-4c57-8c69-91310ce6e283" path="/var/lib/kubelet/pods/4e7814e6-b986-4c57-8c69-91310ce6e283/volumes"
Nov 02 12:24:43.736640 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:43.736590    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d944846-nnbj2"]
Nov 02 12:24:43.740646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:43.740617    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-866d944846-nnbj2"]
Nov 02 12:24:44.234401 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:44.234367    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:44.234817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:44.234780    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:44.235526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:44.235506    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:24:45.583253 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:45.583220    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac" path="/var/lib/kubelet/pods/8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac/volumes"
Nov 02 12:24:51.649173 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:51.649135    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:51.669296 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:51.669265    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:52.960010 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:52.958514    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:24:58.348893 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:24:58.348851    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:24:59.975230 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:24:59.975194    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"]
Nov 02 12:25:00.026251 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:00.026217    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1cc3a7b3-p46h8"]
Nov 02 12:25:01.269765 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.269729    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"]
Nov 02 12:25:01.270416 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.270393    4464 cpu_manager.go:414] "containerMap: removing container" podUID="8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac" containerName="payload-processing"
Nov 02 12:25:01.270416 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.270415    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac" containerName="payload-processing"
Nov 02 12:25:01.270628 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.270606    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3e5450-8dbf-4abe-a6fd-bdeab5c359ac" containerName="payload-processing"
Nov 02 12:25:01.276291 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.276270    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.339938 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.339905    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9cbdfaf1-openshift-default-dockercfg-vjlsz\""
Nov 02 12:25:01.415889 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.415825    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"]
Nov 02 12:25:01.493031 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.468926    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-token\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.494233 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.494205    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-data\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.494436 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.494416    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-workload-certs\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.494567 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.494544    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pssxp\" (UniqueName: \"kubernetes.io/projected/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-kube-api-access-pssxp\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.494784 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.494761    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istiod-ca-cert\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.494866 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.494849    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-envoy\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.494959 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.494940    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-credential-socket\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.495156 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.495123    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-podinfo\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.495526 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.495502    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-workload-socket\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.597051 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.596901    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-podinfo\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.597396 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.597373    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-workload-socket\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.597692 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.597673    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-token\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.597857 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.597841    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-data\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.598006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.597969    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-workload-certs\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.598128 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.598114    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pssxp\" (UniqueName: \"kubernetes.io/projected/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-kube-api-access-pssxp\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.598283 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.598265    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istiod-ca-cert\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.598549 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.598428    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-envoy\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.622001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.598733    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-credential-socket\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.622480 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.608782    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-credential-socket\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.622666 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.612101    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-workload-certs\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.622760 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.612363    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-workload-socket\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.622843 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.612880    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-data\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.622940 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.613502    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istiod-ca-cert\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.642731 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.642699    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-podinfo\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.643159 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.643134    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-envoy\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.653741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.653584    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5" path="/var/lib/kubelet/pods/2c2e0964-1eaf-4f6a-92dc-fbfb2ba31df5/volumes"
Nov 02 12:25:01.674275 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.674239    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-token\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.674404 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.674388    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pssxp\" (UniqueName: \"kubernetes.io/projected/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-kube-api-access-pssxp\") pod \"e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") " pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:01.936858 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:01.936781    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:02.377528 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:02.377500    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"]
Nov 02 12:25:02.378509 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:25:02.378464    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef20ec9_0312_461a_8a8b_9f4a1c848e06.slice/crio-a5360db260b5a518b7754cd68a1a73325bf3bd7a01482da756db97dd162fea13 WatchSource:0}: Error finding container a5360db260b5a518b7754cd68a1a73325bf3bd7a01482da756db97dd162fea13: Status 404 returned error can't find the container with id a5360db260b5a518b7754cd68a1a73325bf3bd7a01482da756db97dd162fea13
Nov 02 12:25:02.380936 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:02.380906    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:25:02.381065 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:02.380969    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:25:02.381065 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:02.381032    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:25:02.903820 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:02.903780    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl" event={"ID":"5ef20ec9-0312-461a-8a8b-9f4a1c848e06","Type":"ContainerStarted","Data":"975b73ae8d69aa812a62e8b07c5742ef5bfa38307995b3b61cea4e780f184ec4"}
Nov 02 12:25:02.904027 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:02.903843    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl" event={"ID":"5ef20ec9-0312-461a-8a8b-9f4a1c848e06","Type":"ContainerStarted","Data":"a5360db260b5a518b7754cd68a1a73325bf3bd7a01482da756db97dd162fea13"}
Nov 02 12:25:02.936890 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:02.936859    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:03.156955 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:03.156860    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl" podStartSLOduration=2.156848045 podStartE2EDuration="2.156848045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:25:03.105025735 +0000 UTC m=+1731.668734147" watchObservedRunningTime="2026-11-02 12:25:03.156848045 +0000 UTC m=+1731.720556453"
Nov 02 12:25:03.942502 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:03.942470    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:04.919968 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:04.919939    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:04.920821 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:04.920799    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:25:09.040999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.036100    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"]
Nov 02 12:25:09.060088 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.053731    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"
Nov 02 12:25:09.135318 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.119420    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"]
Nov 02 12:25:09.156553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.156518    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9cbdfaf1\""
Nov 02 12:25:09.157291 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.157239    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9cbdfaf1-dockercfg-248hb\""
Nov 02 12:25:09.158490 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.158443    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69fc3013-760e-45fe-9f22-95862975a088-plugins-config-volume\") pod \"payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq\" (UID: \"69fc3013-760e-45fe-9f22-95862975a088\") " pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"
Nov 02 12:25:09.158769 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.158748    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rdb7\" (UniqueName: \"kubernetes.io/projected/69fc3013-760e-45fe-9f22-95862975a088-kube-api-access-5rdb7\") pod \"payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq\" (UID: \"69fc3013-760e-45fe-9f22-95862975a088\") " pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"
Nov 02 12:25:09.228808 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.228777    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"]
Nov 02 12:25:09.233473 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.233450    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"
Nov 02 12:25:09.260470 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.260447    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzgw2\" (UniqueName: \"kubernetes.io/projected/9178f68c-8750-48c6-9c3b-2f2a27a62f71-kube-api-access-rzgw2\") pod \"payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29\" (UID: \"9178f68c-8750-48c6-9c3b-2f2a27a62f71\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"
Nov 02 12:25:09.260584 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.260488    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69fc3013-760e-45fe-9f22-95862975a088-plugins-config-volume\") pod \"payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq\" (UID: \"69fc3013-760e-45fe-9f22-95862975a088\") " pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"
Nov 02 12:25:09.260637 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.260589    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9178f68c-8750-48c6-9c3b-2f2a27a62f71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29\" (UID: \"9178f68c-8750-48c6-9c3b-2f2a27a62f71\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"
Nov 02 12:25:09.260715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.260702    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rdb7\" (UniqueName: \"kubernetes.io/projected/69fc3013-760e-45fe-9f22-95862975a088-kube-api-access-5rdb7\") pod \"payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq\" (UID: \"69fc3013-760e-45fe-9f22-95862975a088\") " pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"
Nov 02 12:25:09.261366 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.261345    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69fc3013-760e-45fe-9f22-95862975a088-plugins-config-volume\") pod \"payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq\" (UID: \"69fc3013-760e-45fe-9f22-95862975a088\") " pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"
Nov 02 12:25:09.291842 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.291783    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"]
Nov 02 12:25:09.323366 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.323335    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rdb7\" (UniqueName: \"kubernetes.io/projected/69fc3013-760e-45fe-9f22-95862975a088-kube-api-access-5rdb7\") pod \"payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq\" (UID: \"69fc3013-760e-45fe-9f22-95862975a088\") " pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"
Nov 02 12:25:09.362291 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.362261    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzgw2\" (UniqueName: \"kubernetes.io/projected/9178f68c-8750-48c6-9c3b-2f2a27a62f71-kube-api-access-rzgw2\") pod \"payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29\" (UID: \"9178f68c-8750-48c6-9c3b-2f2a27a62f71\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"
Nov 02 12:25:09.362415 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.362361    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9178f68c-8750-48c6-9c3b-2f2a27a62f71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29\" (UID: \"9178f68c-8750-48c6-9c3b-2f2a27a62f71\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"
Nov 02 12:25:09.363246 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.363225    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9178f68c-8750-48c6-9c3b-2f2a27a62f71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29\" (UID: \"9178f68c-8750-48c6-9c3b-2f2a27a62f71\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"
Nov 02 12:25:09.386137 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.386110    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"
Nov 02 12:25:09.424433 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.424401    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzgw2\" (UniqueName: \"kubernetes.io/projected/9178f68c-8750-48c6-9c3b-2f2a27a62f71-kube-api-access-rzgw2\") pod \"payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29\" (UID: \"9178f68c-8750-48c6-9c3b-2f2a27a62f71\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"
Nov 02 12:25:09.547488 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.547404    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"
Nov 02 12:25:09.763423 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:25:09.763391    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69fc3013_760e_45fe_9f22_95862975a088.slice/crio-c0ee72e3ad7713c8aa744e488ef858d6e0d5d8e6c569768f0badd094575a6421 WatchSource:0}: Error finding container c0ee72e3ad7713c8aa744e488ef858d6e0d5d8e6c569768f0badd094575a6421: Status 404 returned error can't find the container with id c0ee72e3ad7713c8aa744e488ef858d6e0d5d8e6c569768f0badd094575a6421
Nov 02 12:25:09.764169 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.764147    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"]
Nov 02 12:25:09.923019 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:25:09.922956    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9178f68c_8750_48c6_9c3b_2f2a27a62f71.slice/crio-6cc89f9bd254935e1c88ee2a2e2859b2ee200c59ecfd1631df5525bbc6b77876 WatchSource:0}: Error finding container 6cc89f9bd254935e1c88ee2a2e2859b2ee200c59ecfd1631df5525bbc6b77876: Status 404 returned error can't find the container with id 6cc89f9bd254935e1c88ee2a2e2859b2ee200c59ecfd1631df5525bbc6b77876
Nov 02 12:25:09.923935 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.923905    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"]
Nov 02 12:25:09.978903 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.978875    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq" event={"ID":"69fc3013-760e-45fe-9f22-95862975a088","Type":"ContainerStarted","Data":"93de602f705496928dd075a3f546016d2a74dd6d6d6f050226bab680345694af"}
Nov 02 12:25:09.979055 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.978925    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq" event={"ID":"69fc3013-760e-45fe-9f22-95862975a088","Type":"ContainerStarted","Data":"c0ee72e3ad7713c8aa744e488ef858d6e0d5d8e6c569768f0badd094575a6421"}
Nov 02 12:25:09.979128 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.979104    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"
Nov 02 12:25:09.980405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:09.980376    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29" event={"ID":"9178f68c-8750-48c6-9c3b-2f2a27a62f71","Type":"ContainerStarted","Data":"6cc89f9bd254935e1c88ee2a2e2859b2ee200c59ecfd1631df5525bbc6b77876"}
Nov 02 12:25:10.052952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:10.052879    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"]
Nov 02 12:25:10.072772 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:10.072728    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07b38599-whwmv"]
Nov 02 12:25:10.099619 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:10.099561    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq" podStartSLOduration=2.099549539 podStartE2EDuration="2.099549539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:25:10.098668826 +0000 UTC m=+1738.662377236" watchObservedRunningTime="2026-11-02 12:25:10.099549539 +0000 UTC m=+1738.663257947"
Nov 02 12:25:10.989956 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:10.989920    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29" event={"ID":"9178f68c-8750-48c6-9c3b-2f2a27a62f71","Type":"ContainerStarted","Data":"bea6d0a1dc8dcb2a5f0b177cb6ecae386a660d292485774eadd85f772cd96fb2"}
Nov 02 12:25:10.990184 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:10.990025    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"
Nov 02 12:25:11.107887 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:11.107833    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29" podStartSLOduration=2.107819056 podStartE2EDuration="2.107819056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:25:11.106692719 +0000 UTC m=+1739.670401154" watchObservedRunningTime="2026-11-02 12:25:11.107819056 +0000 UTC m=+1739.671527467"
Nov 02 12:25:11.428693 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:11.428609    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:25:11.582883 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:11.582842    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="931ead32-c95e-43a9-b721-d3725cc713f4" path="/var/lib/kubelet/pods/931ead32-c95e-43a9-b721-d3725cc713f4/volumes"
Nov 02 12:25:12.761319 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:12.761289    4464 scope.go:122] "RemoveContainer" containerID="33537237feeb0a10977bd5f2283e05516d59ae62694dd39211d121d37c841b14"
Nov 02 12:25:12.771987 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:12.771956    4464 scope.go:122] "RemoveContainer" containerID="4eba4344af5b9231041c889af30ca39d23d2255a525f3b66cefb69ca4a3754d2"
Nov 02 12:25:12.781084 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:12.781064    4464 scope.go:122] "RemoveContainer" containerID="2887d628f26e26e236fd7eb89f2f199be7917f0fba3a3f4ab1a076cb376d10ff"
Nov 02 12:25:12.790697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:12.790678    4464 scope.go:122] "RemoveContainer" containerID="3212566d8920e7ad968e770cdbcb4da810c3c778991b81fdc9c4249f5c57b171"
Nov 02 12:25:12.800258 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:12.800235    4464 scope.go:122] "RemoveContainer" containerID="128df30c7bfd99963e7f729fc79274fd6393101d08e8d7a9dbe61da9f8b77226"
Nov 02 12:25:20.993091 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:20.993057    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"
Nov 02 12:25:22.008754 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:22.008722    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"
Nov 02 12:25:26.051029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:26.050923    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"]
Nov 02 12:25:26.101596 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:26.101549    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"]
Nov 02 12:25:26.105191 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:26.105170    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-613d7352-pq8m7"]
Nov 02 12:25:26.108349 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:26.108320    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cc8e74b-fc5fj"]
Nov 02 12:25:27.583449 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:27.583414    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f69f9ba-c4ef-4444-b74f-b2d7f16c8852" path="/var/lib/kubelet/pods/1f69f9ba-c4ef-4444-b74f-b2d7f16c8852/volumes"
Nov 02 12:25:27.583871 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:27.583857    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33dbb82b-2601-4e00-883c-2b463feea237" path="/var/lib/kubelet/pods/33dbb82b-2601-4e00-883c-2b463feea237/volumes"
Nov 02 12:25:28.292442 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:25:28.292402    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:25:37.052289 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:37.052254    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"]
Nov 02 12:25:37.102594 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:37.102552    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00690f22-7mfxc"]
Nov 02 12:25:37.582349 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:37.582320    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7" path="/var/lib/kubelet/pods/f1d1a43e-ac53-4ccf-9028-1d9dd232b0e7/volumes"
Nov 02 12:25:41.048941 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:41.048857    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"]
Nov 02 12:25:41.105017 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:41.100907    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7fe11b7-lqtjh"]
Nov 02 12:25:41.582611 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:41.582581    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e05943db-8502-44e9-a7de-4c3a635d2055" path="/var/lib/kubelet/pods/e05943db-8502-44e9-a7de-4c3a635d2055/volumes"
Nov 02 12:25:56.050469 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:56.050429    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"]
Nov 02 12:25:56.100806 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:56.100773    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3635de3-72ccf"]
Nov 02 12:25:57.583316 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:57.583284    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8552f81-8b94-449f-9a45-db0690ae890e" path="/var/lib/kubelet/pods/d8552f81-8b94-449f-9a45-db0690ae890e/volumes"
Nov 02 12:25:58.293047 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:25:58.293010    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:25:58.352051 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:25:58.352023    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:26:08.478729 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:08.478695    4464 generic.go:356] "Generic (PLEG): container finished" podID="5ef20ec9-0312-461a-8a8b-9f4a1c848e06" containerID="975b73ae8d69aa812a62e8b07c5742ef5bfa38307995b3b61cea4e780f184ec4" exitCode=0
Nov 02 12:26:08.479180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:08.478774    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl" event={"ID":"5ef20ec9-0312-461a-8a8b-9f4a1c848e06","Type":"ContainerDied","Data":"975b73ae8d69aa812a62e8b07c5742ef5bfa38307995b3b61cea4e780f184ec4"}
Nov 02 12:26:08.479349 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:08.479327    4464 scope.go:122] "RemoveContainer" containerID="975b73ae8d69aa812a62e8b07c5742ef5bfa38307995b3b61cea4e780f184ec4"
Nov 02 12:26:08.481695 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:08.481648    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:26:08.481822 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:08.481799    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:26:08.481914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:08.481893    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:26:09.491409 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:09.491374    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl" event={"ID":"5ef20ec9-0312-461a-8a8b-9f4a1c848e06","Type":"ContainerStarted","Data":"400832eb320cc52dddee05dfcf8771a8f2a5617fe7fafbb87278605f3b034b3d"}
Nov 02 12:26:09.938439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:09.938347    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:26:09.938822 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:09.938711    4464 patch_prober.go:29] interesting pod/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 02 12:26:09.939055 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:09.939020    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl" podUID="5ef20ec9-0312-461a-8a8b-9f4a1c848e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 02 12:26:10.937641 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:10.937610    4464 patch_prober.go:29] interesting pod/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 02 12:26:10.938033 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:10.937670    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl" podUID="5ef20ec9-0312-461a-8a8b-9f4a1c848e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 02 12:26:11.832510 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:11.832482    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:26:11.834867 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:11.834846    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:26:11.848290 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:11.848266    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:26:11.851916 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:11.851891    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:26:11.937327 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:11.937296    4464 patch_prober.go:29] interesting pod/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 02 12:26:11.937461 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:11.937347    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl" podUID="5ef20ec9-0312-461a-8a8b-9f4a1c848e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 02 12:26:12.887470 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:12.887436    4464 scope.go:122] "RemoveContainer" containerID="347f04988127f4fcc7ec63da51fbaa60920ae555bce1064fe4d1d4725784b194"
Nov 02 12:26:12.903534 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:12.903498    4464 scope.go:122] "RemoveContainer" containerID="b689441e33d3512ccfe36097ea8963aad7520b1e7d7f9b608e7a082814ac2a5e"
Nov 02 12:26:12.913796 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:12.913759    4464 scope.go:122] "RemoveContainer" containerID="a7c321d6f8d22b1ba5c53f54a05682639c28eba1251b9a932f96aa0100a07192"
Nov 02 12:26:12.923139 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:12.923119    4464 scope.go:122] "RemoveContainer" containerID="37e330362c10e1c1a650a3ed635a1f946ce769f8be2cc1fe2fc2e9c00a4804c7"
Nov 02 12:26:12.932273 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:12.932253    4464 scope.go:122] "RemoveContainer" containerID="20319900b15ec34ecb65bbe8d753418726ed0454a411ba38f722b21fea5d0a22"
Nov 02 12:26:12.937550 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:12.937525    4464 patch_prober.go:29] interesting pod/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 02 12:26:12.938380 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:12.937596    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl" podUID="5ef20ec9-0312-461a-8a8b-9f4a1c848e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 02 12:26:13.937688 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:13.937654    4464 patch_prober.go:29] interesting pod/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Nov 02 12:26:13.938111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:13.937712    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl" podUID="5ef20ec9-0312-461a-8a8b-9f4a1c848e06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Nov 02 12:26:14.940833 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:14.940804    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:26:14.941321 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:14.941303    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:26:14.941919 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:14.941898    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:26:21.054881 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:21.054845    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"]
Nov 02 12:26:21.105263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:21.105226    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f88e997-t979s"]
Nov 02 12:26:21.587158 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:21.584306    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd1e6985-ca21-41cb-9bb4-442b25174e9f" path="/var/lib/kubelet/pods/fd1e6985-ca21-41cb-9bb4-442b25174e9f/volumes"
Nov 02 12:26:23.150697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:23.150665    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"]
Nov 02 12:26:23.258182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:23.258152    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"]
Nov 02 12:26:28.151745 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.151705    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k" podUID="e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f" containerName="payload-processing" containerID="cri-o://0074432e965b2065248ad025df88b1f82fb31b45e02966a04f3bd8786d74d027" gracePeriod=25
Nov 02 12:26:28.259191 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.259113    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt" podUID="57ff1019-3226-4cdc-adf8-65d3ac66ab70" containerName="payload-pre-processing" containerID="cri-o://4eb9275c988516617c2ce30708c8efa9eb4856b2233ff3fdfde004cf1e9d5c74" gracePeriod=25
Nov 02 12:26:28.293053 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:26:28.293014    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:26:28.433777 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.433735    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"
Nov 02 12:26:28.478789 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.478753    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f-kube-api-access-5s7d2\" (UniqueName: \"kubernetes.io/projected/e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f-kube-api-access-5s7d2\") pod \"e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f\" (UID: \"e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f\") "
Nov 02 12:26:28.478942 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.478841    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f-plugins-config-volume\") pod \"e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f\" (UID: \"e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f\") "
Nov 02 12:26:28.479471 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.479442    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f-plugins-config-volume" pod "e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f" (UID: "e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:26:28.481201 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.481170    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f-kube-api-access-5s7d2" pod "e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f" (UID: "e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f"). InnerVolumeSpecName "kube-api-access-5s7d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:26:28.525180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.525155    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"
Nov 02 12:26:28.580559 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.580521    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57ff1019-3226-4cdc-adf8-65d3ac66ab70-kube-api-access-v6tnk\" (UniqueName: \"kubernetes.io/projected/57ff1019-3226-4cdc-adf8-65d3ac66ab70-kube-api-access-v6tnk\") pod \"57ff1019-3226-4cdc-adf8-65d3ac66ab70\" (UID: \"57ff1019-3226-4cdc-adf8-65d3ac66ab70\") "
Nov 02 12:26:28.580748 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.580587    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57ff1019-3226-4cdc-adf8-65d3ac66ab70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ff1019-3226-4cdc-adf8-65d3ac66ab70-plugins-config-volume\") pod \"57ff1019-3226-4cdc-adf8-65d3ac66ab70\" (UID: \"57ff1019-3226-4cdc-adf8-65d3ac66ab70\") "
Nov 02 12:26:28.580960 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.580937    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ff1019-3226-4cdc-adf8-65d3ac66ab70-plugins-config-volume" pod "57ff1019-3226-4cdc-adf8-65d3ac66ab70" (UID: "57ff1019-3226-4cdc-adf8-65d3ac66ab70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:26:28.581123 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.581106    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s7d2\" (UniqueName: \"kubernetes.io/projected/e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f-kube-api-access-5s7d2\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:26:28.581191 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.581125    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ff1019-3226-4cdc-adf8-65d3ac66ab70-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:26:28.581191 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.581137    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:26:28.582740 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.582713    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ff1019-3226-4cdc-adf8-65d3ac66ab70-kube-api-access-v6tnk" pod "57ff1019-3226-4cdc-adf8-65d3ac66ab70" (UID: "57ff1019-3226-4cdc-adf8-65d3ac66ab70"). InnerVolumeSpecName "kube-api-access-v6tnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:26:28.682381 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.682310    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6tnk\" (UniqueName: \"kubernetes.io/projected/57ff1019-3226-4cdc-adf8-65d3ac66ab70-kube-api-access-v6tnk\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:26:28.709140 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.708948    4464 generic.go:356] "Generic (PLEG): container finished" podID="57ff1019-3226-4cdc-adf8-65d3ac66ab70" containerID="4eb9275c988516617c2ce30708c8efa9eb4856b2233ff3fdfde004cf1e9d5c74" exitCode=0
Nov 02 12:26:28.709347 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.709190    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt" event={"ID":"57ff1019-3226-4cdc-adf8-65d3ac66ab70","Type":"ContainerDied","Data":"4eb9275c988516617c2ce30708c8efa9eb4856b2233ff3fdfde004cf1e9d5c74"}
Nov 02 12:26:28.709347 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.709253    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt" event={"ID":"57ff1019-3226-4cdc-adf8-65d3ac66ab70","Type":"ContainerDied","Data":"299636b10b9aca888c13b265afdad26344c0a9a882c97d8ecee76a71dc62f266"}
Nov 02 12:26:28.709347 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.709301    4464 scope.go:122] "RemoveContainer" containerID="4eb9275c988516617c2ce30708c8efa9eb4856b2233ff3fdfde004cf1e9d5c74"
Nov 02 12:26:28.709521 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.709474    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"
Nov 02 12:26:28.712154 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.712125    4464 generic.go:356] "Generic (PLEG): container finished" podID="e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f" containerID="0074432e965b2065248ad025df88b1f82fb31b45e02966a04f3bd8786d74d027" exitCode=0
Nov 02 12:26:28.712281 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.712197    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"
Nov 02 12:26:28.712365 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.712346    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k" event={"ID":"e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f","Type":"ContainerDied","Data":"0074432e965b2065248ad025df88b1f82fb31b45e02966a04f3bd8786d74d027"}
Nov 02 12:26:28.712426 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.712400    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k" event={"ID":"e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f","Type":"ContainerDied","Data":"4c5b5571fdcd530e6420af9b3d2b2dcac6bd213b4c5b1bf03a13f42c61cb64ca"}
Nov 02 12:26:28.722883 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.722861    4464 scope.go:122] "RemoveContainer" containerID="4eb9275c988516617c2ce30708c8efa9eb4856b2233ff3fdfde004cf1e9d5c74"
Nov 02 12:26:28.723224 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:26:28.723199    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eb9275c988516617c2ce30708c8efa9eb4856b2233ff3fdfde004cf1e9d5c74\": container with ID starting with 4eb9275c988516617c2ce30708c8efa9eb4856b2233ff3fdfde004cf1e9d5c74 not found: ID does not exist" containerID="4eb9275c988516617c2ce30708c8efa9eb4856b2233ff3fdfde004cf1e9d5c74"
Nov 02 12:26:28.723314 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.723237    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eb9275c988516617c2ce30708c8efa9eb4856b2233ff3fdfde004cf1e9d5c74"} err="failed to get container status \"4eb9275c988516617c2ce30708c8efa9eb4856b2233ff3fdfde004cf1e9d5c74\": rpc error: code = NotFound desc = could not find container \"4eb9275c988516617c2ce30708c8efa9eb4856b2233ff3fdfde004cf1e9d5c74\": container with ID starting with 4eb9275c988516617c2ce30708c8efa9eb4856b2233ff3fdfde004cf1e9d5c74 not found: ID does not exist"
Nov 02 12:26:28.723314 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.723263    4464 scope.go:122] "RemoveContainer" containerID="0074432e965b2065248ad025df88b1f82fb31b45e02966a04f3bd8786d74d027"
Nov 02 12:26:28.733432 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.733410    4464 scope.go:122] "RemoveContainer" containerID="0074432e965b2065248ad025df88b1f82fb31b45e02966a04f3bd8786d74d027"
Nov 02 12:26:28.733691 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:26:28.733669    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0074432e965b2065248ad025df88b1f82fb31b45e02966a04f3bd8786d74d027\": container with ID starting with 0074432e965b2065248ad025df88b1f82fb31b45e02966a04f3bd8786d74d027 not found: ID does not exist" containerID="0074432e965b2065248ad025df88b1f82fb31b45e02966a04f3bd8786d74d027"
Nov 02 12:26:28.733775 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.733698    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0074432e965b2065248ad025df88b1f82fb31b45e02966a04f3bd8786d74d027"} err="failed to get container status \"0074432e965b2065248ad025df88b1f82fb31b45e02966a04f3bd8786d74d027\": rpc error: code = NotFound desc = could not find container \"0074432e965b2065248ad025df88b1f82fb31b45e02966a04f3bd8786d74d027\": container with ID starting with 0074432e965b2065248ad025df88b1f82fb31b45e02966a04f3bd8786d74d027 not found: ID does not exist"
Nov 02 12:26:28.878913 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.878881    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"]
Nov 02 12:26:28.885077 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.885052    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6a432197-6d85858967-wmttt"]
Nov 02 12:26:28.995161 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.995133    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"]
Nov 02 12:26:28.999298 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:28.999273    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6a432197-749f9ff54f-txt7k"]
Nov 02 12:26:29.582386 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:29.582352    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57ff1019-3226-4cdc-adf8-65d3ac66ab70" path="/var/lib/kubelet/pods/57ff1019-3226-4cdc-adf8-65d3ac66ab70/volumes"
Nov 02 12:26:29.582802 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:29.582787    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f" path="/var/lib/kubelet/pods/e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f/volumes"
Nov 02 12:26:31.384126 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:31.384093    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:26:37.048827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:37.048790    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"]
Nov 02 12:26:37.099036 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:37.099010    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d40ee7f-kkthl"]
Nov 02 12:26:37.582657 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:37.582616    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5" path="/var/lib/kubelet/pods/b3cc04a4-dcc8-4eec-a73c-ed10d7867cb5/volumes"
Nov 02 12:26:38.628115 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.628083    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"]
Nov 02 12:26:38.628714 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.628699    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f" containerName="payload-processing"
Nov 02 12:26:38.628755 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.628715    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f" containerName="payload-processing"
Nov 02 12:26:38.628755 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.628745    4464 cpu_manager.go:414] "containerMap: removing container" podUID="57ff1019-3226-4cdc-adf8-65d3ac66ab70" containerName="payload-pre-processing"
Nov 02 12:26:38.628755 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.628751    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57ff1019-3226-4cdc-adf8-65d3ac66ab70" containerName="payload-pre-processing"
Nov 02 12:26:38.628865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.628857    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23de9bc-9894-4a21-8e6f-c60b3b9c8e9f" containerName="payload-processing"
Nov 02 12:26:38.628895 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.628872    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="57ff1019-3226-4cdc-adf8-65d3ac66ab70" containerName="payload-pre-processing"
Nov 02 12:26:38.632403 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.632386    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"
Nov 02 12:26:38.690404 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.690376    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"]
Nov 02 12:26:38.712360 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.712331    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c20863-752d-448c-901f-214a073e3395-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd\" (UID: \"70c20863-752d-448c-901f-214a073e3395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"
Nov 02 12:26:38.712496 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.712383    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xqm2\" (UniqueName: \"kubernetes.io/projected/70c20863-752d-448c-901f-214a073e3395-kube-api-access-8xqm2\") pod \"maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd\" (UID: \"70c20863-752d-448c-901f-214a073e3395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"
Nov 02 12:26:38.712634 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.712606    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c20863-752d-448c-901f-214a073e3395-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd\" (UID: \"70c20863-752d-448c-901f-214a073e3395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"
Nov 02 12:26:38.736277 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.736246    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:26:38.736277 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.736254    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:26:38.736449 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.736302    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:26:38.736449 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.736378    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:26:38.813493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.813459    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c20863-752d-448c-901f-214a073e3395-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd\" (UID: \"70c20863-752d-448c-901f-214a073e3395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"
Nov 02 12:26:38.813629 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.813613    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c20863-752d-448c-901f-214a073e3395-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd\" (UID: \"70c20863-752d-448c-901f-214a073e3395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"
Nov 02 12:26:38.813726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.813663    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xqm2\" (UniqueName: \"kubernetes.io/projected/70c20863-752d-448c-901f-214a073e3395-kube-api-access-8xqm2\") pod \"maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd\" (UID: \"70c20863-752d-448c-901f-214a073e3395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"
Nov 02 12:26:38.814319 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.814292    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c20863-752d-448c-901f-214a073e3395-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd\" (UID: \"70c20863-752d-448c-901f-214a073e3395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"
Nov 02 12:26:38.816245 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.816221    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c20863-752d-448c-901f-214a073e3395-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd\" (UID: \"70c20863-752d-448c-901f-214a073e3395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"
Nov 02 12:26:38.875377 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.875353    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xqm2\" (UniqueName: \"kubernetes.io/projected/70c20863-752d-448c-901f-214a073e3395-kube-api-access-8xqm2\") pod \"maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd\" (UID: \"70c20863-752d-448c-901f-214a073e3395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"
Nov 02 12:26:38.945559 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:38.945495    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"
Nov 02 12:26:39.321821 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:39.321769    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"]
Nov 02 12:26:39.322061 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:26:39.322034    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c20863_752d_448c_901f_214a073e3395.slice/crio-86edadecbfc27ab7d74ec2ab9a153d2cd8fd5e69b409cd1c9ef76f73816c952a WatchSource:0}: Error finding container 86edadecbfc27ab7d74ec2ab9a153d2cd8fd5e69b409cd1c9ef76f73816c952a: Status 404 returned error can't find the container with id 86edadecbfc27ab7d74ec2ab9a153d2cd8fd5e69b409cd1c9ef76f73816c952a
Nov 02 12:26:39.798196 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:39.798164    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd" event={"ID":"70c20863-752d-448c-901f-214a073e3395","Type":"ContainerStarted","Data":"3913346a456238d019b0f309bac91d898e29d710b6e46b815dc4c5a146738623"}
Nov 02 12:26:39.798684 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:39.798229    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd" event={"ID":"70c20863-752d-448c-901f-214a073e3395","Type":"ContainerStarted","Data":"86edadecbfc27ab7d74ec2ab9a153d2cd8fd5e69b409cd1c9ef76f73816c952a"}
Nov 02 12:26:39.917785 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:39.917687    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd" podStartSLOduration=1.917672789 podStartE2EDuration="1.917672789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:39.91496377 +0000 UTC m=+1828.478672179" watchObservedRunningTime="2026-11-02 12:26:39.917672789 +0000 UTC m=+1828.481381207"
Nov 02 12:26:40.806571 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:40.806529    4464 generic.go:356] "Generic (PLEG): container finished" podID="70c20863-752d-448c-901f-214a073e3395" containerID="3913346a456238d019b0f309bac91d898e29d710b6e46b815dc4c5a146738623" exitCode=0
Nov 02 12:26:40.807001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:40.806589    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd" event={"ID":"70c20863-752d-448c-901f-214a073e3395","Type":"ContainerDied","Data":"3913346a456238d019b0f309bac91d898e29d710b6e46b815dc4c5a146738623"}
Nov 02 12:26:41.964311 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:41.964271    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"
Nov 02 12:26:42.066055 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:42.065961    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70c20863-752d-448c-901f-214a073e3395-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c20863-752d-448c-901f-214a073e3395-maas-api-openshift-ca\") pod \"70c20863-752d-448c-901f-214a073e3395\" (UID: \"70c20863-752d-448c-901f-214a073e3395\") "
Nov 02 12:26:42.066194 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:42.066111    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c20863-752d-448c-901f-214a073e3395-kube-api-access-8xqm2\" (UniqueName: \"kubernetes.io/projected/70c20863-752d-448c-901f-214a073e3395-kube-api-access-8xqm2\") pod \"70c20863-752d-448c-901f-214a073e3395\" (UID: \"70c20863-752d-448c-901f-214a073e3395\") "
Nov 02 12:26:42.066299 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:42.066276    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70c20863-752d-448c-901f-214a073e3395-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c20863-752d-448c-901f-214a073e3395-maas-api-xks-ca\") pod \"70c20863-752d-448c-901f-214a073e3395\" (UID: \"70c20863-752d-448c-901f-214a073e3395\") "
Nov 02 12:26:42.066425 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:42.066330    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c20863-752d-448c-901f-214a073e3395-maas-api-openshift-ca" pod "70c20863-752d-448c-901f-214a073e3395" (UID: "70c20863-752d-448c-901f-214a073e3395"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:26:42.066682 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:42.066663    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c20863-752d-448c-901f-214a073e3395-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:26:42.068165 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:42.068142    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c20863-752d-448c-901f-214a073e3395-maas-api-xks-ca" pod "70c20863-752d-448c-901f-214a073e3395" (UID: "70c20863-752d-448c-901f-214a073e3395"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:26:42.068246 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:42.068229    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c20863-752d-448c-901f-214a073e3395-kube-api-access-8xqm2" pod "70c20863-752d-448c-901f-214a073e3395" (UID: "70c20863-752d-448c-901f-214a073e3395"). InnerVolumeSpecName "kube-api-access-8xqm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:26:42.168113 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:42.168076    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xqm2\" (UniqueName: \"kubernetes.io/projected/70c20863-752d-448c-901f-214a073e3395-kube-api-access-8xqm2\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:26:42.168113 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:42.168109    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c20863-752d-448c-901f-214a073e3395-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:26:42.823136 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:42.823100    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd" event={"ID":"70c20863-752d-448c-901f-214a073e3395","Type":"ContainerDied","Data":"86edadecbfc27ab7d74ec2ab9a153d2cd8fd5e69b409cd1c9ef76f73816c952a"}
Nov 02 12:26:42.823389 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:42.823142    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"
Nov 02 12:26:42.823389 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:42.823154    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86edadecbfc27ab7d74ec2ab9a153d2cd8fd5e69b409cd1c9ef76f73816c952a"
Nov 02 12:26:50.131956 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:50.131919    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"]
Nov 02 12:26:50.249039 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:50.249004    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"]
Nov 02 12:26:51.055521 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:51.055491    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"]
Nov 02 12:26:51.058984 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:51.058946    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ef7c53-ktlpg"]
Nov 02 12:26:51.592063 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:51.592027    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aedbeb68-0cf8-48c4-a5e8-780c5db76876" path="/var/lib/kubelet/pods/aedbeb68-0cf8-48c4-a5e8-780c5db76876/volumes"
Nov 02 12:26:55.132724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.132639    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq" podUID="69fc3013-760e-45fe-9f22-95862975a088" containerName="payload-processing" containerID="cri-o://93de602f705496928dd075a3f546016d2a74dd6d6d6f050226bab680345694af" gracePeriod=25
Nov 02 12:26:55.249655 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.249624    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29" podUID="9178f68c-8750-48c6-9c3b-2f2a27a62f71" containerName="payload-pre-processing" containerID="cri-o://bea6d0a1dc8dcb2a5f0b177cb6ecae386a660d292485774eadd85f772cd96fb2" gracePeriod=25
Nov 02 12:26:55.399505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.399435    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"
Nov 02 12:26:55.488456 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.488425    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fc3013-760e-45fe-9f22-95862975a088-kube-api-access-5rdb7\" (UniqueName: \"kubernetes.io/projected/69fc3013-760e-45fe-9f22-95862975a088-kube-api-access-5rdb7\") pod \"69fc3013-760e-45fe-9f22-95862975a088\" (UID: \"69fc3013-760e-45fe-9f22-95862975a088\") "
Nov 02 12:26:55.488817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.488681    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69fc3013-760e-45fe-9f22-95862975a088-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69fc3013-760e-45fe-9f22-95862975a088-plugins-config-volume\") pod \"69fc3013-760e-45fe-9f22-95862975a088\" (UID: \"69fc3013-760e-45fe-9f22-95862975a088\") "
Nov 02 12:26:55.489368 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.489336    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fc3013-760e-45fe-9f22-95862975a088-plugins-config-volume" pod "69fc3013-760e-45fe-9f22-95862975a088" (UID: "69fc3013-760e-45fe-9f22-95862975a088"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:26:55.491001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.490958    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fc3013-760e-45fe-9f22-95862975a088-kube-api-access-5rdb7" pod "69fc3013-760e-45fe-9f22-95862975a088" (UID: "69fc3013-760e-45fe-9f22-95862975a088"). InnerVolumeSpecName "kube-api-access-5rdb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:26:55.508497 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.508474    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"
Nov 02 12:26:55.589738 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.589711    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9178f68c-8750-48c6-9c3b-2f2a27a62f71-kube-api-access-rzgw2\" (UniqueName: \"kubernetes.io/projected/9178f68c-8750-48c6-9c3b-2f2a27a62f71-kube-api-access-rzgw2\") pod \"9178f68c-8750-48c6-9c3b-2f2a27a62f71\" (UID: \"9178f68c-8750-48c6-9c3b-2f2a27a62f71\") "
Nov 02 12:26:55.589912 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.589860    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9178f68c-8750-48c6-9c3b-2f2a27a62f71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9178f68c-8750-48c6-9c3b-2f2a27a62f71-plugins-config-volume\") pod \"9178f68c-8750-48c6-9c3b-2f2a27a62f71\" (UID: \"9178f68c-8750-48c6-9c3b-2f2a27a62f71\") "
Nov 02 12:26:55.590255 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.590214    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69fc3013-760e-45fe-9f22-95862975a088-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:26:55.590255 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.590241    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rdb7\" (UniqueName: \"kubernetes.io/projected/69fc3013-760e-45fe-9f22-95862975a088-kube-api-access-5rdb7\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:26:55.590398 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.590292    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9178f68c-8750-48c6-9c3b-2f2a27a62f71-plugins-config-volume" pod "9178f68c-8750-48c6-9c3b-2f2a27a62f71" (UID: "9178f68c-8750-48c6-9c3b-2f2a27a62f71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:26:55.591946 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.591925    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9178f68c-8750-48c6-9c3b-2f2a27a62f71-kube-api-access-rzgw2" pod "9178f68c-8750-48c6-9c3b-2f2a27a62f71" (UID: "9178f68c-8750-48c6-9c3b-2f2a27a62f71"). InnerVolumeSpecName "kube-api-access-rzgw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:26:55.691945 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.691869    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9178f68c-8750-48c6-9c3b-2f2a27a62f71-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:26:55.691945 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.691901    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzgw2\" (UniqueName: \"kubernetes.io/projected/9178f68c-8750-48c6-9c3b-2f2a27a62f71-kube-api-access-rzgw2\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:26:55.970714 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.970683    4464 generic.go:356] "Generic (PLEG): container finished" podID="69fc3013-760e-45fe-9f22-95862975a088" containerID="93de602f705496928dd075a3f546016d2a74dd6d6d6f050226bab680345694af" exitCode=0
Nov 02 12:26:55.970882 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.970742    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"
Nov 02 12:26:55.970882 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.970771    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq" event={"ID":"69fc3013-760e-45fe-9f22-95862975a088","Type":"ContainerDied","Data":"93de602f705496928dd075a3f546016d2a74dd6d6d6f050226bab680345694af"}
Nov 02 12:26:55.970882 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.970861    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq" event={"ID":"69fc3013-760e-45fe-9f22-95862975a088","Type":"ContainerDied","Data":"c0ee72e3ad7713c8aa744e488ef858d6e0d5d8e6c569768f0badd094575a6421"}
Nov 02 12:26:55.971032 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.970898    4464 scope.go:122] "RemoveContainer" containerID="93de602f705496928dd075a3f546016d2a74dd6d6d6f050226bab680345694af"
Nov 02 12:26:55.972621 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.972596    4464 generic.go:356] "Generic (PLEG): container finished" podID="9178f68c-8750-48c6-9c3b-2f2a27a62f71" containerID="bea6d0a1dc8dcb2a5f0b177cb6ecae386a660d292485774eadd85f772cd96fb2" exitCode=0
Nov 02 12:26:55.972735 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.972631    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29" event={"ID":"9178f68c-8750-48c6-9c3b-2f2a27a62f71","Type":"ContainerDied","Data":"bea6d0a1dc8dcb2a5f0b177cb6ecae386a660d292485774eadd85f772cd96fb2"}
Nov 02 12:26:55.972735 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.972645    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"
Nov 02 12:26:55.972735 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.972673    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29" event={"ID":"9178f68c-8750-48c6-9c3b-2f2a27a62f71","Type":"ContainerDied","Data":"6cc89f9bd254935e1c88ee2a2e2859b2ee200c59ecfd1631df5525bbc6b77876"}
Nov 02 12:26:55.981342 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.981326    4464 scope.go:122] "RemoveContainer" containerID="93de602f705496928dd075a3f546016d2a74dd6d6d6f050226bab680345694af"
Nov 02 12:26:55.981584 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:26:55.981568    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93de602f705496928dd075a3f546016d2a74dd6d6d6f050226bab680345694af\": container with ID starting with 93de602f705496928dd075a3f546016d2a74dd6d6d6f050226bab680345694af not found: ID does not exist" containerID="93de602f705496928dd075a3f546016d2a74dd6d6d6f050226bab680345694af"
Nov 02 12:26:55.981631 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.981591    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93de602f705496928dd075a3f546016d2a74dd6d6d6f050226bab680345694af"} err="failed to get container status \"93de602f705496928dd075a3f546016d2a74dd6d6d6f050226bab680345694af\": rpc error: code = NotFound desc = could not find container \"93de602f705496928dd075a3f546016d2a74dd6d6d6f050226bab680345694af\": container with ID starting with 93de602f705496928dd075a3f546016d2a74dd6d6d6f050226bab680345694af not found: ID does not exist"
Nov 02 12:26:55.981631 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.981605    4464 scope.go:122] "RemoveContainer" containerID="bea6d0a1dc8dcb2a5f0b177cb6ecae386a660d292485774eadd85f772cd96fb2"
Nov 02 12:26:55.991300 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.991281    4464 scope.go:122] "RemoveContainer" containerID="bea6d0a1dc8dcb2a5f0b177cb6ecae386a660d292485774eadd85f772cd96fb2"
Nov 02 12:26:55.991571 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:26:55.991553    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea6d0a1dc8dcb2a5f0b177cb6ecae386a660d292485774eadd85f772cd96fb2\": container with ID starting with bea6d0a1dc8dcb2a5f0b177cb6ecae386a660d292485774eadd85f772cd96fb2 not found: ID does not exist" containerID="bea6d0a1dc8dcb2a5f0b177cb6ecae386a660d292485774eadd85f772cd96fb2"
Nov 02 12:26:55.991620 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:55.991578    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea6d0a1dc8dcb2a5f0b177cb6ecae386a660d292485774eadd85f772cd96fb2"} err="failed to get container status \"bea6d0a1dc8dcb2a5f0b177cb6ecae386a660d292485774eadd85f772cd96fb2\": rpc error: code = NotFound desc = could not find container \"bea6d0a1dc8dcb2a5f0b177cb6ecae386a660d292485774eadd85f772cd96fb2\": container with ID starting with bea6d0a1dc8dcb2a5f0b177cb6ecae386a660d292485774eadd85f772cd96fb2 not found: ID does not exist"
Nov 02 12:26:56.146221 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:56.146185    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"]
Nov 02 12:26:56.151728 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:56.151703    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9cbdfaf1-58867fd66c-l6spq"]
Nov 02 12:26:56.258742 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:56.258665    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"]
Nov 02 12:26:56.262787 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:56.262759    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9cbdfaf1-97865bdbc-gkn29"]
Nov 02 12:26:57.583185 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:57.583155    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69fc3013-760e-45fe-9f22-95862975a088" path="/var/lib/kubelet/pods/69fc3013-760e-45fe-9f22-95862975a088/volumes"
Nov 02 12:26:57.583598 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:26:57.583580    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9178f68c-8750-48c6-9c3b-2f2a27a62f71" path="/var/lib/kubelet/pods/9178f68c-8750-48c6-9c3b-2f2a27a62f71/volumes"
Nov 02 12:26:58.292875 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:26:58.292838    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:27:01.049584 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:01.049553    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"]
Nov 02 12:27:01.100304 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:01.100264    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17be0c70-9rn78"]
Nov 02 12:27:01.582674 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:01.582639    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583a7279-0e1d-4542-8db7-946602fa9b37" path="/var/lib/kubelet/pods/583a7279-0e1d-4542-8db7-946602fa9b37/volumes"
Nov 02 12:27:04.340254 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:04.340210    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"]
Nov 02 12:27:04.340756 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:04.340459    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d" podUID="e834d8c0-89ce-4797-b500-fd61f84ad58e" containerName="istio-proxy" containerID="cri-o://bd2ec375f5adc3f7d24a3d6465bda54519f28cb2e8a1c1505170805cff48a648" gracePeriod=30
Nov 02 12:27:08.551294 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:08.551255    4464 patch_prober.go:29] interesting pod/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:27:08.551719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:08.551318    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d" podUID="e834d8c0-89ce-4797-b500-fd61f84ad58e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:27:09.607625 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.607554    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:27:09.712395 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.712371    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-token\" (UniqueName: \"kubernetes.io/projected/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-token\") pod \"e834d8c0-89ce-4797-b500-fd61f84ad58e\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") "
Nov 02 12:27:09.712560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.712407    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e834d8c0-89ce-4797-b500-fd61f84ad58e-kube-api-access-bwnww\" (UniqueName: \"kubernetes.io/projected/e834d8c0-89ce-4797-b500-fd61f84ad58e-kube-api-access-bwnww\") pod \"e834d8c0-89ce-4797-b500-fd61f84ad58e\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") "
Nov 02 12:27:09.712560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.712436    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-envoy\") pod \"e834d8c0-89ce-4797-b500-fd61f84ad58e\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") "
Nov 02 12:27:09.712560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.712474    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-data\") pod \"e834d8c0-89ce-4797-b500-fd61f84ad58e\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") "
Nov 02 12:27:09.712560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.712530    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-workload-socket\") pod \"e834d8c0-89ce-4797-b500-fd61f84ad58e\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") "
Nov 02 12:27:09.712560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.712553    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-podinfo\") pod \"e834d8c0-89ce-4797-b500-fd61f84ad58e\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") "
Nov 02 12:27:09.712855 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.712694    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-credential-socket\") pod \"e834d8c0-89ce-4797-b500-fd61f84ad58e\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") "
Nov 02 12:27:09.712855 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.712736    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e834d8c0-89ce-4797-b500-fd61f84ad58e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e834d8c0-89ce-4797-b500-fd61f84ad58e-istiod-ca-cert\") pod \"e834d8c0-89ce-4797-b500-fd61f84ad58e\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") "
Nov 02 12:27:09.712855 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.712776    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-workload-certs\") pod \"e834d8c0-89ce-4797-b500-fd61f84ad58e\" (UID: \"e834d8c0-89ce-4797-b500-fd61f84ad58e\") "
Nov 02 12:27:09.713024 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.712852    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-data" pod "e834d8c0-89ce-4797-b500-fd61f84ad58e" (UID: "e834d8c0-89ce-4797-b500-fd61f84ad58e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:27:09.713383 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.713104    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-credential-socket" pod "e834d8c0-89ce-4797-b500-fd61f84ad58e" (UID: "e834d8c0-89ce-4797-b500-fd61f84ad58e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:27:09.713383 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.713208    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-workload-certs" pod "e834d8c0-89ce-4797-b500-fd61f84ad58e" (UID: "e834d8c0-89ce-4797-b500-fd61f84ad58e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:27:09.713383 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.713231    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e834d8c0-89ce-4797-b500-fd61f84ad58e-istiod-ca-cert" pod "e834d8c0-89ce-4797-b500-fd61f84ad58e" (UID: "e834d8c0-89ce-4797-b500-fd61f84ad58e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:09.713383 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.713328    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-workload-socket" pod "e834d8c0-89ce-4797-b500-fd61f84ad58e" (UID: "e834d8c0-89ce-4797-b500-fd61f84ad58e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:27:09.713661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.713565    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:09.713661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.713587    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e834d8c0-89ce-4797-b500-fd61f84ad58e-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:09.713661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.713603    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:09.713661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.713617    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:09.713661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.713631    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:09.715086 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.715060    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-token" pod "e834d8c0-89ce-4797-b500-fd61f84ad58e" (UID: "e834d8c0-89ce-4797-b500-fd61f84ad58e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:09.715086 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.715076    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e834d8c0-89ce-4797-b500-fd61f84ad58e-kube-api-access-bwnww" pod "e834d8c0-89ce-4797-b500-fd61f84ad58e" (UID: "e834d8c0-89ce-4797-b500-fd61f84ad58e"). InnerVolumeSpecName "kube-api-access-bwnww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:09.715219 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.715137    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-podinfo" pod "e834d8c0-89ce-4797-b500-fd61f84ad58e" (UID: "e834d8c0-89ce-4797-b500-fd61f84ad58e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:27:09.716104 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.716084    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-envoy" pod "e834d8c0-89ce-4797-b500-fd61f84ad58e" (UID: "e834d8c0-89ce-4797-b500-fd61f84ad58e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:27:09.815079 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.815050    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:09.815079 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.815078    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:09.815263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.815089    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwnww\" (UniqueName: \"kubernetes.io/projected/e834d8c0-89ce-4797-b500-fd61f84ad58e-kube-api-access-bwnww\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:09.815263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:09.815100    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e834d8c0-89ce-4797-b500-fd61f84ad58e-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:10.074469 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:10.074431    4464 generic.go:356] "Generic (PLEG): container finished" podID="e834d8c0-89ce-4797-b500-fd61f84ad58e" containerID="bd2ec375f5adc3f7d24a3d6465bda54519f28cb2e8a1c1505170805cff48a648" exitCode=0
Nov 02 12:27:10.074678 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:10.074488    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"
Nov 02 12:27:10.074678 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:10.074516    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d" event={"ID":"e834d8c0-89ce-4797-b500-fd61f84ad58e","Type":"ContainerDied","Data":"bd2ec375f5adc3f7d24a3d6465bda54519f28cb2e8a1c1505170805cff48a648"}
Nov 02 12:27:10.074678 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:10.074576    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d" event={"ID":"e834d8c0-89ce-4797-b500-fd61f84ad58e","Type":"ContainerDied","Data":"88769a8185a030b5d19a27a71a9f64dddd2e370c7e96b51e57f6f21fc49366ec"}
Nov 02 12:27:10.074678 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:10.074603    4464 scope.go:122] "RemoveContainer" containerID="bd2ec375f5adc3f7d24a3d6465bda54519f28cb2e8a1c1505170805cff48a648"
Nov 02 12:27:10.086397 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:10.086380    4464 scope.go:122] "RemoveContainer" containerID="bd2ec375f5adc3f7d24a3d6465bda54519f28cb2e8a1c1505170805cff48a648"
Nov 02 12:27:10.086676 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:27:10.086659    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd2ec375f5adc3f7d24a3d6465bda54519f28cb2e8a1c1505170805cff48a648\": container with ID starting with bd2ec375f5adc3f7d24a3d6465bda54519f28cb2e8a1c1505170805cff48a648 not found: ID does not exist" containerID="bd2ec375f5adc3f7d24a3d6465bda54519f28cb2e8a1c1505170805cff48a648"
Nov 02 12:27:10.086731 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:10.086684    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd2ec375f5adc3f7d24a3d6465bda54519f28cb2e8a1c1505170805cff48a648"} err="failed to get container status \"bd2ec375f5adc3f7d24a3d6465bda54519f28cb2e8a1c1505170805cff48a648\": rpc error: code = NotFound desc = could not find container \"bd2ec375f5adc3f7d24a3d6465bda54519f28cb2e8a1c1505170805cff48a648\": container with ID starting with bd2ec375f5adc3f7d24a3d6465bda54519f28cb2e8a1c1505170805cff48a648 not found: ID does not exist"
Nov 02 12:27:10.244130 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:10.244092    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"]
Nov 02 12:27:10.294407 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:10.294372    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6a432197-openshift-default-6c5bc96b97-p6c8d"]
Nov 02 12:27:11.306835 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.306803    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"]
Nov 02 12:27:11.307435 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.307420    4464 cpu_manager.go:414] "containerMap: removing container" podUID="70c20863-752d-448c-901f-214a073e3395" containerName="revoke-keys"
Nov 02 12:27:11.307482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.307437    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c20863-752d-448c-901f-214a073e3395" containerName="revoke-keys"
Nov 02 12:27:11.307482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.307462    4464 cpu_manager.go:414] "containerMap: removing container" podUID="9178f68c-8750-48c6-9c3b-2f2a27a62f71" containerName="payload-pre-processing"
Nov 02 12:27:11.307482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.307468    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9178f68c-8750-48c6-9c3b-2f2a27a62f71" containerName="payload-pre-processing"
Nov 02 12:27:11.307592 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.307493    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e834d8c0-89ce-4797-b500-fd61f84ad58e" containerName="istio-proxy"
Nov 02 12:27:11.307592 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.307498    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e834d8c0-89ce-4797-b500-fd61f84ad58e" containerName="istio-proxy"
Nov 02 12:27:11.307592 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.307509    4464 cpu_manager.go:414] "containerMap: removing container" podUID="69fc3013-760e-45fe-9f22-95862975a088" containerName="payload-processing"
Nov 02 12:27:11.307592 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.307514    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fc3013-760e-45fe-9f22-95862975a088" containerName="payload-processing"
Nov 02 12:27:11.307760 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.307596    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e834d8c0-89ce-4797-b500-fd61f84ad58e" containerName="istio-proxy"
Nov 02 12:27:11.307760 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.307607    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fc3013-760e-45fe-9f22-95862975a088" containerName="payload-processing"
Nov 02 12:27:11.307760 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.307616    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="70c20863-752d-448c-901f-214a073e3395" containerName="revoke-keys"
Nov 02 12:27:11.307760 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.307624    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="9178f68c-8750-48c6-9c3b-2f2a27a62f71" containerName="payload-pre-processing"
Nov 02 12:27:11.312547 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.312529    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"
Nov 02 12:27:11.366110 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.366086    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:27:11.369489 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.369467    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"]
Nov 02 12:27:11.416104 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.416067    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:27:11.418155 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.416330    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:27:11.451230 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.451178    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc45a848-4783-4184-aab0-965d9e164047-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eba33e27-jwkkz\" (UID: \"cc45a848-4783-4184-aab0-965d9e164047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"
Nov 02 12:27:11.451794 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.451250    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpq6j\" (UniqueName: \"kubernetes.io/projected/cc45a848-4783-4184-aab0-965d9e164047-kube-api-access-bpq6j\") pod \"maas-api-revoke-sub-eba33e27-jwkkz\" (UID: \"cc45a848-4783-4184-aab0-965d9e164047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"
Nov 02 12:27:11.451794 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.451346    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc45a848-4783-4184-aab0-965d9e164047-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eba33e27-jwkkz\" (UID: \"cc45a848-4783-4184-aab0-965d9e164047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"
Nov 02 12:27:11.466224 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.466184    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:27:11.553811 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.553771    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc45a848-4783-4184-aab0-965d9e164047-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eba33e27-jwkkz\" (UID: \"cc45a848-4783-4184-aab0-965d9e164047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"
Nov 02 12:27:11.582043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.564903    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc45a848-4783-4184-aab0-965d9e164047-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eba33e27-jwkkz\" (UID: \"cc45a848-4783-4184-aab0-965d9e164047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"
Nov 02 12:27:11.673877 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.673840    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpq6j\" (UniqueName: \"kubernetes.io/projected/cc45a848-4783-4184-aab0-965d9e164047-kube-api-access-bpq6j\") pod \"maas-api-revoke-sub-eba33e27-jwkkz\" (UID: \"cc45a848-4783-4184-aab0-965d9e164047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"
Nov 02 12:27:11.674720 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.635304    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:27:11.674904 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.664247    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc45a848-4783-4184-aab0-965d9e164047-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eba33e27-jwkkz\" (UID: \"cc45a848-4783-4184-aab0-965d9e164047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"
Nov 02 12:27:11.675084 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.610908    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc45a848-4783-4184-aab0-965d9e164047-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eba33e27-jwkkz\" (UID: \"cc45a848-4783-4184-aab0-965d9e164047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"
Nov 02 12:27:11.726302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.726256    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e834d8c0-89ce-4797-b500-fd61f84ad58e" path="/var/lib/kubelet/pods/e834d8c0-89ce-4797-b500-fd61f84ad58e/volumes"
Nov 02 12:27:11.789476 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.789446    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:27:11.799856 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.799833    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpq6j\" (UniqueName: \"kubernetes.io/projected/cc45a848-4783-4184-aab0-965d9e164047-kube-api-access-bpq6j\") pod \"maas-api-revoke-sub-eba33e27-jwkkz\" (UID: \"cc45a848-4783-4184-aab0-965d9e164047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"
Nov 02 12:27:11.977765 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.977735    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:27:11.985964 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:11.985945    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"
Nov 02 12:27:12.386968 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:27:12.386940    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc45a848_4783_4184_aab0_965d9e164047.slice/crio-3287e9602277fa97635c48140927d4f23db6ae2db1d019782d0c6e2a92ac2f32 WatchSource:0}: Error finding container 3287e9602277fa97635c48140927d4f23db6ae2db1d019782d0c6e2a92ac2f32: Status 404 returned error can't find the container with id 3287e9602277fa97635c48140927d4f23db6ae2db1d019782d0c6e2a92ac2f32
Nov 02 12:27:12.387417 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:12.387396    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"]
Nov 02 12:27:12.651295 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:12.651215    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:27:13.005470 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:13.005443    4464 scope.go:122] "RemoveContainer" containerID="3506d814c124158e8968f4ed6637214c093593adc92441a29a945f5c515d53ac"
Nov 02 12:27:13.015132 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:13.015114    4464 scope.go:122] "RemoveContainer" containerID="4f4b58b98b105b7f18569c113ef8ab1567fb37b6c63394d24e0acdec8734c9c1"
Nov 02 12:27:13.023803 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:13.023779    4464 scope.go:122] "RemoveContainer" containerID="a7dd4588aa990846e6ec2235e7bc034372f6eff1a72b59649cbfdc479d9426d0"
Nov 02 12:27:13.032014 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:13.031997    4464 scope.go:122] "RemoveContainer" containerID="e01e3ad6dc9f3d4301cf53dc9bf1ad456e28369b045c4b3f9eb89f2b3d1d652a"
Nov 02 12:27:13.041045 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:13.040919    4464 scope.go:122] "RemoveContainer" containerID="7789532192a1afdbf2178293a8e7d4856567af4013b93e302fab696a67113228"
Nov 02 12:27:13.105032 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:13.105005    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz" event={"ID":"cc45a848-4783-4184-aab0-965d9e164047","Type":"ContainerStarted","Data":"a398e23b1c536ac06573e4d794e0ee1a67683f5fcb1aaaea7162c95156ee7d01"}
Nov 02 12:27:13.105130 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:13.105055    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz" event={"ID":"cc45a848-4783-4184-aab0-965d9e164047","Type":"ContainerStarted","Data":"3287e9602277fa97635c48140927d4f23db6ae2db1d019782d0c6e2a92ac2f32"}
Nov 02 12:27:13.368887 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:13.368832    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz" podStartSLOduration=2.368817288 podStartE2EDuration="2.368817288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:13.227453616 +0000 UTC m=+1861.791162051" watchObservedRunningTime="2026-11-02 12:27:13.368817288 +0000 UTC m=+1861.932525745"
Nov 02 12:27:13.369629 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:13.369609    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"]
Nov 02 12:27:13.372852 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:13.372825    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9cbdfaf1-hb4qd"]
Nov 02 12:27:13.647262 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:13.645171    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70c20863-752d-448c-901f-214a073e3395" path="/var/lib/kubelet/pods/70c20863-752d-448c-901f-214a073e3395/volumes"
Nov 02 12:27:14.118199 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:14.118159    4464 generic.go:356] "Generic (PLEG): container finished" podID="cc45a848-4783-4184-aab0-965d9e164047" containerID="a398e23b1c536ac06573e4d794e0ee1a67683f5fcb1aaaea7162c95156ee7d01" exitCode=0
Nov 02 12:27:14.118438 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:14.118405    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz" event={"ID":"cc45a848-4783-4184-aab0-965d9e164047","Type":"ContainerDied","Data":"a398e23b1c536ac06573e4d794e0ee1a67683f5fcb1aaaea7162c95156ee7d01"}
Nov 02 12:27:14.850342 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:14.850285    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"]
Nov 02 12:27:14.851316 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:14.851274    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl" podUID="5ef20ec9-0312-461a-8a8b-9f4a1c848e06" containerName="istio-proxy" containerID="cri-o://400832eb320cc52dddee05dfcf8771a8f2a5617fe7fafbb87278605f3b034b3d" gracePeriod=30
Nov 02 12:27:15.068355 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:15.066880    4464 patch_prober.go:29] interesting pod/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:27:15.068355 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:15.066956    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl" podUID="5ef20ec9-0312-461a-8a8b-9f4a1c848e06" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:27:15.460911 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:15.460880    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"
Nov 02 12:27:15.556654 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:15.555437    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc45a848-4783-4184-aab0-965d9e164047-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc45a848-4783-4184-aab0-965d9e164047-maas-api-openshift-ca\") pod \"cc45a848-4783-4184-aab0-965d9e164047\" (UID: \"cc45a848-4783-4184-aab0-965d9e164047\") "
Nov 02 12:27:15.556654 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:15.555567    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc45a848-4783-4184-aab0-965d9e164047-kube-api-access-bpq6j\" (UniqueName: \"kubernetes.io/projected/cc45a848-4783-4184-aab0-965d9e164047-kube-api-access-bpq6j\") pod \"cc45a848-4783-4184-aab0-965d9e164047\" (UID: \"cc45a848-4783-4184-aab0-965d9e164047\") "
Nov 02 12:27:15.556654 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:15.555649    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc45a848-4783-4184-aab0-965d9e164047-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc45a848-4783-4184-aab0-965d9e164047-maas-api-xks-ca\") pod \"cc45a848-4783-4184-aab0-965d9e164047\" (UID: \"cc45a848-4783-4184-aab0-965d9e164047\") "
Nov 02 12:27:15.557676 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:15.557356    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc45a848-4783-4184-aab0-965d9e164047-maas-api-openshift-ca" pod "cc45a848-4783-4184-aab0-965d9e164047" (UID: "cc45a848-4783-4184-aab0-965d9e164047"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:15.560449 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:15.560410    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc45a848-4783-4184-aab0-965d9e164047-maas-api-xks-ca" pod "cc45a848-4783-4184-aab0-965d9e164047" (UID: "cc45a848-4783-4184-aab0-965d9e164047"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:15.602289 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:15.602251    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc45a848-4783-4184-aab0-965d9e164047-kube-api-access-bpq6j" pod "cc45a848-4783-4184-aab0-965d9e164047" (UID: "cc45a848-4783-4184-aab0-965d9e164047"). InnerVolumeSpecName "kube-api-access-bpq6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:15.723072 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:15.703479    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc45a848-4783-4184-aab0-965d9e164047-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:15.723072 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:15.703519    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpq6j\" (UniqueName: \"kubernetes.io/projected/cc45a848-4783-4184-aab0-965d9e164047-kube-api-access-bpq6j\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:15.723072 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:15.703536    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc45a848-4783-4184-aab0-965d9e164047-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:16.401066 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:16.401021    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz" event={"ID":"cc45a848-4783-4184-aab0-965d9e164047","Type":"ContainerDied","Data":"3287e9602277fa97635c48140927d4f23db6ae2db1d019782d0c6e2a92ac2f32"}
Nov 02 12:27:16.401749 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:16.401727    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3287e9602277fa97635c48140927d4f23db6ae2db1d019782d0c6e2a92ac2f32"
Nov 02 12:27:16.402022 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:16.402003    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"
Nov 02 12:27:17.244109 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:17.244069    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lwhz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:27:17.244885 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:17.244852    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4" podUID="eb51d042-8930-45f9-89c9-fe9ad351df24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:27:17.755000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:17.754356    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-62g8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:27:17.755000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:17.754420    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c" podUID="896e3cac-30cf-492b-a20a-cec437dc6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:27:17.779000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:17.777055    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jthbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:27:17.779000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:17.777111    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv" podUID="890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:27:19.053042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.052987    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"]
Nov 02 12:27:19.104627 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.104586    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b8494e-hj2xj"]
Nov 02 12:27:19.582811 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.582778    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c2aeb0d-0494-4743-8b75-dfe3cfb568af" path="/var/lib/kubelet/pods/8c2aeb0d-0494-4743-8b75-dfe3cfb568af/volumes"
Nov 02 12:27:19.733867 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.733825    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"]
Nov 02 12:27:19.734497 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.734482    4464 cpu_manager.go:414] "containerMap: removing container" podUID="cc45a848-4783-4184-aab0-965d9e164047" containerName="revoke-keys"
Nov 02 12:27:19.734593 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.734498    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc45a848-4783-4184-aab0-965d9e164047" containerName="revoke-keys"
Nov 02 12:27:19.734634 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.734611    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc45a848-4783-4184-aab0-965d9e164047" containerName="revoke-keys"
Nov 02 12:27:19.738241 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.738225    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"
Nov 02 12:27:19.790837 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.790796    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3463300d-10e1-4637-b554-e9f3a34f9a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfe8bfdd-t6pbs\" (UID: \"3463300d-10e1-4637-b554-e9f3a34f9a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"
Nov 02 12:27:19.791008 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.790861    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3463300d-10e1-4637-b554-e9f3a34f9a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfe8bfdd-t6pbs\" (UID: \"3463300d-10e1-4637-b554-e9f3a34f9a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"
Nov 02 12:27:19.791008 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.790911    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"]
Nov 02 12:27:19.791204 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.791050    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpw62\" (UniqueName: \"kubernetes.io/projected/3463300d-10e1-4637-b554-e9f3a34f9a6b-kube-api-access-fpw62\") pod \"maas-api-revoke-sub-cfe8bfdd-t6pbs\" (UID: \"3463300d-10e1-4637-b554-e9f3a34f9a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"
Nov 02 12:27:19.791514 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.791494    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:27:19.837111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.837037    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:27:19.837111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.837034    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:27:19.837111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.837077    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:27:19.892641 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.892617    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpw62\" (UniqueName: \"kubernetes.io/projected/3463300d-10e1-4637-b554-e9f3a34f9a6b-kube-api-access-fpw62\") pod \"maas-api-revoke-sub-cfe8bfdd-t6pbs\" (UID: \"3463300d-10e1-4637-b554-e9f3a34f9a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"
Nov 02 12:27:19.892768 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.892753    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3463300d-10e1-4637-b554-e9f3a34f9a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfe8bfdd-t6pbs\" (UID: \"3463300d-10e1-4637-b554-e9f3a34f9a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"
Nov 02 12:27:19.892811 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.892793    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3463300d-10e1-4637-b554-e9f3a34f9a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfe8bfdd-t6pbs\" (UID: \"3463300d-10e1-4637-b554-e9f3a34f9a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"
Nov 02 12:27:19.893456 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.893434    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3463300d-10e1-4637-b554-e9f3a34f9a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfe8bfdd-t6pbs\" (UID: \"3463300d-10e1-4637-b554-e9f3a34f9a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"
Nov 02 12:27:19.895315 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.895293    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3463300d-10e1-4637-b554-e9f3a34f9a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfe8bfdd-t6pbs\" (UID: \"3463300d-10e1-4637-b554-e9f3a34f9a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"
Nov 02 12:27:19.954799 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:19.954771    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpw62\" (UniqueName: \"kubernetes.io/projected/3463300d-10e1-4637-b554-e9f3a34f9a6b-kube-api-access-fpw62\") pod \"maas-api-revoke-sub-cfe8bfdd-t6pbs\" (UID: \"3463300d-10e1-4637-b554-e9f3a34f9a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"
Nov 02 12:27:20.052561 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.052541    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"
Nov 02 12:27:20.084064 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.084036    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:27:20.095324 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.095242    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-podinfo\") pod \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") "
Nov 02 12:27:20.095459 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.095377    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-envoy\") pod \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") "
Nov 02 12:27:20.095459 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.095438    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-credential-socket\") pod \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") "
Nov 02 12:27:20.095573 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.095487    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istiod-ca-cert\") pod \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") "
Nov 02 12:27:20.095573 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.095550    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-workload-socket\") pod \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") "
Nov 02 12:27:20.095772 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.095748    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-token\" (UniqueName: \"kubernetes.io/projected/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-token\") pod \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") "
Nov 02 12:27:20.095842 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.095789    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-kube-api-access-pssxp\" (UniqueName: \"kubernetes.io/projected/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-kube-api-access-pssxp\") pod \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") "
Nov 02 12:27:20.095842 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.095806    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-workload-socket" pod "5ef20ec9-0312-461a-8a8b-9f4a1c848e06" (UID: "5ef20ec9-0312-461a-8a8b-9f4a1c848e06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:27:20.095951 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.095861    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-data\") pod \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") "
Nov 02 12:27:20.095951 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.095893    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-workload-certs\") pod \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\" (UID: \"5ef20ec9-0312-461a-8a8b-9f4a1c848e06\") "
Nov 02 12:27:20.096135 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.095952    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istiod-ca-cert" pod "5ef20ec9-0312-461a-8a8b-9f4a1c848e06" (UID: "5ef20ec9-0312-461a-8a8b-9f4a1c848e06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:20.096704 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.096664    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-data" pod "5ef20ec9-0312-461a-8a8b-9f4a1c848e06" (UID: "5ef20ec9-0312-461a-8a8b-9f4a1c848e06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:27:20.096916 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.096893    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-workload-certs" pod "5ef20ec9-0312-461a-8a8b-9f4a1c848e06" (UID: "5ef20ec9-0312-461a-8a8b-9f4a1c848e06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:27:20.097009 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.096967    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-credential-socket" pod "5ef20ec9-0312-461a-8a8b-9f4a1c848e06" (UID: "5ef20ec9-0312-461a-8a8b-9f4a1c848e06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:27:20.097192 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.097173    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:20.097285 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.097199    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:20.097285 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.097213    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:20.097285 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.097229    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:20.097285 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.097242    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:20.097524 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.097505    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-envoy" pod "5ef20ec9-0312-461a-8a8b-9f4a1c848e06" (UID: "5ef20ec9-0312-461a-8a8b-9f4a1c848e06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:27:20.097599 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.097585    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-podinfo" pod "5ef20ec9-0312-461a-8a8b-9f4a1c848e06" (UID: "5ef20ec9-0312-461a-8a8b-9f4a1c848e06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:27:20.098166 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.098152    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-kube-api-access-pssxp" pod "5ef20ec9-0312-461a-8a8b-9f4a1c848e06" (UID: "5ef20ec9-0312-461a-8a8b-9f4a1c848e06"). InnerVolumeSpecName "kube-api-access-pssxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:20.098213 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.098164    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-token" pod "5ef20ec9-0312-461a-8a8b-9f4a1c848e06" (UID: "5ef20ec9-0312-461a-8a8b-9f4a1c848e06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:20.198695 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.198664    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:20.198695 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.198695    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:20.198897 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.198704    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:20.198897 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.198714    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pssxp\" (UniqueName: \"kubernetes.io/projected/5ef20ec9-0312-461a-8a8b-9f4a1c848e06-kube-api-access-pssxp\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:20.428659 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.428633    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"]
Nov 02 12:27:20.431273 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:27:20.431242    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3463300d_10e1_4637_b554_e9f3a34f9a6b.slice/crio-2b7f81a6437caa705fa5f56e095b3612e7afa1c79bad4f96fb627373548337f2 WatchSource:0}: Error finding container 2b7f81a6437caa705fa5f56e095b3612e7afa1c79bad4f96fb627373548337f2: Status 404 returned error can't find the container with id 2b7f81a6437caa705fa5f56e095b3612e7afa1c79bad4f96fb627373548337f2
Nov 02 12:27:20.503226 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.503195    4464 generic.go:356] "Generic (PLEG): container finished" podID="5ef20ec9-0312-461a-8a8b-9f4a1c848e06" containerID="400832eb320cc52dddee05dfcf8771a8f2a5617fe7fafbb87278605f3b034b3d" exitCode=0
Nov 02 12:27:20.503352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.503247    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"
Nov 02 12:27:20.503352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.503271    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl" event={"ID":"5ef20ec9-0312-461a-8a8b-9f4a1c848e06","Type":"ContainerDied","Data":"400832eb320cc52dddee05dfcf8771a8f2a5617fe7fafbb87278605f3b034b3d"}
Nov 02 12:27:20.503352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.503334    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl" event={"ID":"5ef20ec9-0312-461a-8a8b-9f4a1c848e06","Type":"ContainerDied","Data":"a5360db260b5a518b7754cd68a1a73325bf3bd7a01482da756db97dd162fea13"}
Nov 02 12:27:20.503517 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.503358    4464 scope.go:122] "RemoveContainer" containerID="400832eb320cc52dddee05dfcf8771a8f2a5617fe7fafbb87278605f3b034b3d"
Nov 02 12:27:20.505175 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.505140    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs" event={"ID":"3463300d-10e1-4637-b554-e9f3a34f9a6b","Type":"ContainerStarted","Data":"2b7f81a6437caa705fa5f56e095b3612e7afa1c79bad4f96fb627373548337f2"}
Nov 02 12:27:20.514242 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.514222    4464 scope.go:122] "RemoveContainer" containerID="975b73ae8d69aa812a62e8b07c5742ef5bfa38307995b3b61cea4e780f184ec4"
Nov 02 12:27:20.524076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.524059    4464 scope.go:122] "RemoveContainer" containerID="400832eb320cc52dddee05dfcf8771a8f2a5617fe7fafbb87278605f3b034b3d"
Nov 02 12:27:20.524364 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:27:20.524343    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400832eb320cc52dddee05dfcf8771a8f2a5617fe7fafbb87278605f3b034b3d\": container with ID starting with 400832eb320cc52dddee05dfcf8771a8f2a5617fe7fafbb87278605f3b034b3d not found: ID does not exist" containerID="400832eb320cc52dddee05dfcf8771a8f2a5617fe7fafbb87278605f3b034b3d"
Nov 02 12:27:20.524454 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.524374    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400832eb320cc52dddee05dfcf8771a8f2a5617fe7fafbb87278605f3b034b3d"} err="failed to get container status \"400832eb320cc52dddee05dfcf8771a8f2a5617fe7fafbb87278605f3b034b3d\": rpc error: code = NotFound desc = could not find container \"400832eb320cc52dddee05dfcf8771a8f2a5617fe7fafbb87278605f3b034b3d\": container with ID starting with 400832eb320cc52dddee05dfcf8771a8f2a5617fe7fafbb87278605f3b034b3d not found: ID does not exist"
Nov 02 12:27:20.524454 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.524392    4464 scope.go:122] "RemoveContainer" containerID="975b73ae8d69aa812a62e8b07c5742ef5bfa38307995b3b61cea4e780f184ec4"
Nov 02 12:27:20.524656 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:27:20.524617    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975b73ae8d69aa812a62e8b07c5742ef5bfa38307995b3b61cea4e780f184ec4\": container with ID starting with 975b73ae8d69aa812a62e8b07c5742ef5bfa38307995b3b61cea4e780f184ec4 not found: ID does not exist" containerID="975b73ae8d69aa812a62e8b07c5742ef5bfa38307995b3b61cea4e780f184ec4"
Nov 02 12:27:20.524699 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.524666    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975b73ae8d69aa812a62e8b07c5742ef5bfa38307995b3b61cea4e780f184ec4"} err="failed to get container status \"975b73ae8d69aa812a62e8b07c5742ef5bfa38307995b3b61cea4e780f184ec4\": rpc error: code = NotFound desc = could not find container \"975b73ae8d69aa812a62e8b07c5742ef5bfa38307995b3b61cea4e780f184ec4\": container with ID starting with 975b73ae8d69aa812a62e8b07c5742ef5bfa38307995b3b61cea4e780f184ec4 not found: ID does not exist"
Nov 02 12:27:20.672185 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.672151    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"]
Nov 02 12:27:20.676672 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:20.676647    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9cbdfaf1-openshift-default-7dbbc655c-fw8dl"]
Nov 02 12:27:21.513908 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:21.513820    4464 generic.go:356] "Generic (PLEG): container finished" podID="3463300d-10e1-4637-b554-e9f3a34f9a6b" containerID="8fa026c52f3d7bc86ac278d184bbd2723445690e6cce34aabc45097222ddef16" exitCode=0
Nov 02 12:27:21.513908 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:21.513883    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs" event={"ID":"3463300d-10e1-4637-b554-e9f3a34f9a6b","Type":"ContainerDied","Data":"8fa026c52f3d7bc86ac278d184bbd2723445690e6cce34aabc45097222ddef16"}
Nov 02 12:27:21.584800 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:21.584770    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef20ec9-0312-461a-8a8b-9f4a1c848e06" path="/var/lib/kubelet/pods/5ef20ec9-0312-461a-8a8b-9f4a1c848e06/volumes"
Nov 02 12:27:22.683855 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:22.683832    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"
Nov 02 12:27:22.730917 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:22.730888    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3463300d-10e1-4637-b554-e9f3a34f9a6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3463300d-10e1-4637-b554-e9f3a34f9a6b-maas-api-openshift-ca\") pod \"3463300d-10e1-4637-b554-e9f3a34f9a6b\" (UID: \"3463300d-10e1-4637-b554-e9f3a34f9a6b\") "
Nov 02 12:27:22.731091 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:22.731044    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3463300d-10e1-4637-b554-e9f3a34f9a6b-kube-api-access-fpw62\" (UniqueName: \"kubernetes.io/projected/3463300d-10e1-4637-b554-e9f3a34f9a6b-kube-api-access-fpw62\") pod \"3463300d-10e1-4637-b554-e9f3a34f9a6b\" (UID: \"3463300d-10e1-4637-b554-e9f3a34f9a6b\") "
Nov 02 12:27:22.731158 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:22.731110    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3463300d-10e1-4637-b554-e9f3a34f9a6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3463300d-10e1-4637-b554-e9f3a34f9a6b-maas-api-xks-ca\") pod \"3463300d-10e1-4637-b554-e9f3a34f9a6b\" (UID: \"3463300d-10e1-4637-b554-e9f3a34f9a6b\") "
Nov 02 12:27:22.731309 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:22.731285    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3463300d-10e1-4637-b554-e9f3a34f9a6b-maas-api-openshift-ca" pod "3463300d-10e1-4637-b554-e9f3a34f9a6b" (UID: "3463300d-10e1-4637-b554-e9f3a34f9a6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:22.731492 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:22.731474    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3463300d-10e1-4637-b554-e9f3a34f9a6b-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:22.733166 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:22.733135    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3463300d-10e1-4637-b554-e9f3a34f9a6b-kube-api-access-fpw62" pod "3463300d-10e1-4637-b554-e9f3a34f9a6b" (UID: "3463300d-10e1-4637-b554-e9f3a34f9a6b"). InnerVolumeSpecName "kube-api-access-fpw62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:22.733268 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:22.733142    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3463300d-10e1-4637-b554-e9f3a34f9a6b-maas-api-xks-ca" pod "3463300d-10e1-4637-b554-e9f3a34f9a6b" (UID: "3463300d-10e1-4637-b554-e9f3a34f9a6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:22.833022 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:22.832917    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpw62\" (UniqueName: \"kubernetes.io/projected/3463300d-10e1-4637-b554-e9f3a34f9a6b-kube-api-access-fpw62\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:22.833022 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:22.832950    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3463300d-10e1-4637-b554-e9f3a34f9a6b-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:23.528387 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:23.528362    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"
Nov 02 12:27:23.528564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:23.528405    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs" event={"ID":"3463300d-10e1-4637-b554-e9f3a34f9a6b","Type":"ContainerDied","Data":"2b7f81a6437caa705fa5f56e095b3612e7afa1c79bad4f96fb627373548337f2"}
Nov 02 12:27:23.528564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:23.528461    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b7f81a6437caa705fa5f56e095b3612e7afa1c79bad4f96fb627373548337f2"
Nov 02 12:27:24.428818 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.428787    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"]
Nov 02 12:27:24.429433 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.429417    4464 cpu_manager.go:414] "containerMap: removing container" podUID="5ef20ec9-0312-461a-8a8b-9f4a1c848e06" containerName="istio-proxy"
Nov 02 12:27:24.429476 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.429435    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef20ec9-0312-461a-8a8b-9f4a1c848e06" containerName="istio-proxy"
Nov 02 12:27:24.429476 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.429472    4464 cpu_manager.go:414] "containerMap: removing container" podUID="3463300d-10e1-4637-b554-e9f3a34f9a6b" containerName="revoke-keys"
Nov 02 12:27:24.429546 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.429478    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3463300d-10e1-4637-b554-e9f3a34f9a6b" containerName="revoke-keys"
Nov 02 12:27:24.429577 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.429568    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="3463300d-10e1-4637-b554-e9f3a34f9a6b" containerName="revoke-keys"
Nov 02 12:27:24.429612 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.429584    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef20ec9-0312-461a-8a8b-9f4a1c848e06" containerName="istio-proxy"
Nov 02 12:27:24.429612 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.429594    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef20ec9-0312-461a-8a8b-9f4a1c848e06" containerName="istio-proxy"
Nov 02 12:27:24.442381 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.442355    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"
Nov 02 12:27:24.456409 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.456384    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqjqt\" (UniqueName: \"kubernetes.io/projected/3aa0c066-d198-4b91-bee6-9859d9b0de9c-kube-api-access-jqjqt\") pod \"payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7\" (UID: \"3aa0c066-d198-4b91-bee6-9859d9b0de9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"
Nov 02 12:27:24.456533 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.456423    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa0c066-d198-4b91-bee6-9859d9b0de9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7\" (UID: \"3aa0c066-d198-4b91-bee6-9859d9b0de9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"
Nov 02 12:27:24.492337 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.492305    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"]
Nov 02 12:27:24.541295 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.541260    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ccee868d-dockercfg-4vft4\""
Nov 02 12:27:24.592465 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.571055    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqjqt\" (UniqueName: \"kubernetes.io/projected/3aa0c066-d198-4b91-bee6-9859d9b0de9c-kube-api-access-jqjqt\") pod \"payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7\" (UID: \"3aa0c066-d198-4b91-bee6-9859d9b0de9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"
Nov 02 12:27:24.592855 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.592760    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa0c066-d198-4b91-bee6-9859d9b0de9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7\" (UID: \"3aa0c066-d198-4b91-bee6-9859d9b0de9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"
Nov 02 12:27:24.594107 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.594067    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa0c066-d198-4b91-bee6-9859d9b0de9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7\" (UID: \"3aa0c066-d198-4b91-bee6-9859d9b0de9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"
Nov 02 12:27:24.594273 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.571609    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ccee868d\""
Nov 02 12:27:24.727047 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.725914    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqjqt\" (UniqueName: \"kubernetes.io/projected/3aa0c066-d198-4b91-bee6-9859d9b0de9c-kube-api-access-jqjqt\") pod \"payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7\" (UID: \"3aa0c066-d198-4b91-bee6-9859d9b0de9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"
Nov 02 12:27:24.756654 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:24.756626    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"
Nov 02 12:27:25.150993 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:25.140889    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"]
Nov 02 12:27:25.150993 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:27:25.141490    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa0c066_d198_4b91_bee6_9859d9b0de9c.slice/crio-ecc7a1be6d7bccb855ac1269e853e76311f30074e5804e1c577612c8dd7de6ac WatchSource:0}: Error finding container ecc7a1be6d7bccb855ac1269e853e76311f30074e5804e1c577612c8dd7de6ac: Status 404 returned error can't find the container with id ecc7a1be6d7bccb855ac1269e853e76311f30074e5804e1c577612c8dd7de6ac
Nov 02 12:27:25.747611 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:25.747571    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7" event={"ID":"3aa0c066-d198-4b91-bee6-9859d9b0de9c","Type":"ContainerStarted","Data":"96081459698f89baee16bde4f047c108adaff0cbc88963396ab935575908c743"}
Nov 02 12:27:25.748072 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:25.747639    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"
Nov 02 12:27:25.748072 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:25.747657    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7" event={"ID":"3aa0c066-d198-4b91-bee6-9859d9b0de9c","Type":"ContainerStarted","Data":"ecc7a1be6d7bccb855ac1269e853e76311f30074e5804e1c577612c8dd7de6ac"}
Nov 02 12:27:25.866691 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:25.866642    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7" podStartSLOduration=1.8666284100000001 podStartE2EDuration="1.86662841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:25.865607672 +0000 UTC m=+1874.429316081" watchObservedRunningTime="2026-11-02 12:27:25.86662841 +0000 UTC m=+1874.430336819"
Nov 02 12:27:28.292187 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:27:28.292148    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:27:28.804179 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:28.804147    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/models-as-a-service_payload-processing-external-model-5cc9844f6-pv7vm_1068cf8a-7fc4-48ea-9522-3fc6e914891a/payload-processing/0.log"
Nov 02 12:27:30.397952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.397915    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"]
Nov 02 12:27:30.398559 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.398533    4464 cpu_manager.go:414] "containerMap: removing container" podUID="5ef20ec9-0312-461a-8a8b-9f4a1c848e06" containerName="istio-proxy"
Nov 02 12:27:30.398559 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.398556    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef20ec9-0312-461a-8a8b-9f4a1c848e06" containerName="istio-proxy"
Nov 02 12:27:30.402348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.402330    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"
Nov 02 12:27:30.456357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.456333    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:27:30.509594 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.509564    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:27:30.509879 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.509861    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:27:30.525317 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.525250    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:27:30.526439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.526310    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl8d2\" (UniqueName: \"kubernetes.io/projected/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-kube-api-access-rl8d2\") pod \"maas-api-revoke-sub-f841c1ed-mnnv5\" (UID: \"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"
Nov 02 12:27:30.526578 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.526521    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f841c1ed-mnnv5\" (UID: \"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"
Nov 02 12:27:30.526657 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.526611    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f841c1ed-mnnv5\" (UID: \"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"
Nov 02 12:27:30.531234 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.531205    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"]
Nov 02 12:27:30.627891 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.627849    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f841c1ed-mnnv5\" (UID: \"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"
Nov 02 12:27:30.628448 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.628419    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl8d2\" (UniqueName: \"kubernetes.io/projected/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-kube-api-access-rl8d2\") pod \"maas-api-revoke-sub-f841c1ed-mnnv5\" (UID: \"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"
Nov 02 12:27:30.628683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.628668    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f841c1ed-mnnv5\" (UID: \"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"
Nov 02 12:27:30.629588 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.629566    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f841c1ed-mnnv5\" (UID: \"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"
Nov 02 12:27:30.633343 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.633322    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f841c1ed-mnnv5\" (UID: \"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"
Nov 02 12:27:30.634835 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.634812    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"]
Nov 02 12:27:30.640030 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.640013    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"
Nov 02 12:27:30.674027 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.674004    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"]
Nov 02 12:27:30.693062 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.693040    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl8d2\" (UniqueName: \"kubernetes.io/projected/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-kube-api-access-rl8d2\") pod \"maas-api-revoke-sub-f841c1ed-mnnv5\" (UID: \"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"
Nov 02 12:27:30.716012 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.715970    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"
Nov 02 12:27:30.729797 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.729768    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026d0a04-126f-4959-94a2-9300876e2bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d7d1db0-vjnmz\" (UID: \"026d0a04-126f-4959-94a2-9300876e2bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"
Nov 02 12:27:30.729905 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.729851    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkxb2\" (UniqueName: \"kubernetes.io/projected/026d0a04-126f-4959-94a2-9300876e2bb1-kube-api-access-vkxb2\") pod \"maas-api-revoke-sub-9d7d1db0-vjnmz\" (UID: \"026d0a04-126f-4959-94a2-9300876e2bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"
Nov 02 12:27:30.730071 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.730045    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026d0a04-126f-4959-94a2-9300876e2bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d7d1db0-vjnmz\" (UID: \"026d0a04-126f-4959-94a2-9300876e2bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"
Nov 02 12:27:30.831773 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.831737    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026d0a04-126f-4959-94a2-9300876e2bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d7d1db0-vjnmz\" (UID: \"026d0a04-126f-4959-94a2-9300876e2bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"
Nov 02 12:27:30.831970 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.831814    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkxb2\" (UniqueName: \"kubernetes.io/projected/026d0a04-126f-4959-94a2-9300876e2bb1-kube-api-access-vkxb2\") pod \"maas-api-revoke-sub-9d7d1db0-vjnmz\" (UID: \"026d0a04-126f-4959-94a2-9300876e2bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"
Nov 02 12:27:30.831970 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.831897    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026d0a04-126f-4959-94a2-9300876e2bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d7d1db0-vjnmz\" (UID: \"026d0a04-126f-4959-94a2-9300876e2bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"
Nov 02 12:27:30.832706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.832683    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026d0a04-126f-4959-94a2-9300876e2bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d7d1db0-vjnmz\" (UID: \"026d0a04-126f-4959-94a2-9300876e2bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"
Nov 02 12:27:30.834330 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.834303    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026d0a04-126f-4959-94a2-9300876e2bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d7d1db0-vjnmz\" (UID: \"026d0a04-126f-4959-94a2-9300876e2bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"
Nov 02 12:27:30.893790 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.893765    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkxb2\" (UniqueName: \"kubernetes.io/projected/026d0a04-126f-4959-94a2-9300876e2bb1-kube-api-access-vkxb2\") pod \"maas-api-revoke-sub-9d7d1db0-vjnmz\" (UID: \"026d0a04-126f-4959-94a2-9300876e2bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"
Nov 02 12:27:30.958464 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:30.958404    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"
Nov 02 12:27:31.096869 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:31.096843    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"]
Nov 02 12:27:31.097441 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:27:31.097418    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ffd0dc_fcde_48f8_8f53_3283f0f0dfb1.slice/crio-bfddf70c10a3a702d24fdc8223a52889f690fab53ca914fa6dafe41d986922f0 WatchSource:0}: Error finding container bfddf70c10a3a702d24fdc8223a52889f690fab53ca914fa6dafe41d986922f0: Status 404 returned error can't find the container with id bfddf70c10a3a702d24fdc8223a52889f690fab53ca914fa6dafe41d986922f0
Nov 02 12:27:31.353174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:31.350243    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"]
Nov 02 12:27:31.813029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:31.812970    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz" event={"ID":"026d0a04-126f-4959-94a2-9300876e2bb1","Type":"ContainerStarted","Data":"9341559b37be17df1785ca6e305c61bcdc179e2ec3c098d0d67f64a5b8e0d7f9"}
Nov 02 12:27:31.813386 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:31.813057    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz" event={"ID":"026d0a04-126f-4959-94a2-9300876e2bb1","Type":"ContainerStarted","Data":"39824f9c7642403c826bfe6c75651d36087d75d47d0769350925104c8251f7ab"}
Nov 02 12:27:31.815039 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:31.815009    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5" event={"ID":"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1","Type":"ContainerStarted","Data":"b1efd3a785f50b8239538001c26a619c0268474b93f4404e765a249e37615c2b"}
Nov 02 12:27:31.815134 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:31.815063    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5" event={"ID":"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1","Type":"ContainerStarted","Data":"bfddf70c10a3a702d24fdc8223a52889f690fab53ca914fa6dafe41d986922f0"}
Nov 02 12:27:31.930363 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:31.930268    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz" podStartSLOduration=1.930253083 podStartE2EDuration="1.930253083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:31.929252728 +0000 UTC m=+1880.492961151" watchObservedRunningTime="2026-11-02 12:27:31.930253083 +0000 UTC m=+1880.493961521"
Nov 02 12:27:32.049964 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:32.049938    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"]
Nov 02 12:27:32.051290 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:32.051243    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5" podStartSLOduration=2.051229111 podStartE2EDuration="2.051229111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:32.048279107 +0000 UTC m=+1880.611987517" watchObservedRunningTime="2026-11-02 12:27:32.051229111 +0000 UTC m=+1880.614937519"
Nov 02 12:27:32.055914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:32.055883    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-352a61b2-rjfpn"]
Nov 02 12:27:32.823085 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:32.823053    4464 generic.go:356] "Generic (PLEG): container finished" podID="026d0a04-126f-4959-94a2-9300876e2bb1" containerID="9341559b37be17df1785ca6e305c61bcdc179e2ec3c098d0d67f64a5b8e0d7f9" exitCode=0
Nov 02 12:27:32.823452 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:32.823135    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz" event={"ID":"026d0a04-126f-4959-94a2-9300876e2bb1","Type":"ContainerDied","Data":"9341559b37be17df1785ca6e305c61bcdc179e2ec3c098d0d67f64a5b8e0d7f9"}
Nov 02 12:27:32.824819 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:32.824793    4464 generic.go:356] "Generic (PLEG): container finished" podID="60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1" containerID="b1efd3a785f50b8239538001c26a619c0268474b93f4404e765a249e37615c2b" exitCode=0
Nov 02 12:27:32.824921 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:32.824838    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5" event={"ID":"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1","Type":"ContainerDied","Data":"b1efd3a785f50b8239538001c26a619c0268474b93f4404e765a249e37615c2b"}
Nov 02 12:27:33.583161 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:33.583130    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cace0f3d-c048-4f89-9fb0-180afb8b468c" path="/var/lib/kubelet/pods/cace0f3d-c048-4f89-9fb0-180afb8b468c/volumes"
Nov 02 12:27:34.006439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.006415    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"
Nov 02 12:27:34.009461 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.009438    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"
Nov 02 12:27:34.050609 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.050582    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"]
Nov 02 12:27:34.077380 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.077358    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-kube-api-access-rl8d2\" (UniqueName: \"kubernetes.io/projected/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-kube-api-access-rl8d2\") pod \"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1\" (UID: \"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1\") "
Nov 02 12:27:34.077493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.077412    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026d0a04-126f-4959-94a2-9300876e2bb1-kube-api-access-vkxb2\" (UniqueName: \"kubernetes.io/projected/026d0a04-126f-4959-94a2-9300876e2bb1-kube-api-access-vkxb2\") pod \"026d0a04-126f-4959-94a2-9300876e2bb1\" (UID: \"026d0a04-126f-4959-94a2-9300876e2bb1\") "
Nov 02 12:27:34.077576 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.077550    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026d0a04-126f-4959-94a2-9300876e2bb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026d0a04-126f-4959-94a2-9300876e2bb1-maas-api-openshift-ca\") pod \"026d0a04-126f-4959-94a2-9300876e2bb1\" (UID: \"026d0a04-126f-4959-94a2-9300876e2bb1\") "
Nov 02 12:27:34.077633 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.077604    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-maas-api-openshift-ca\") pod \"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1\" (UID: \"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1\") "
Nov 02 12:27:34.077694 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.077632    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-maas-api-xks-ca\") pod \"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1\" (UID: \"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1\") "
Nov 02 12:27:34.077694 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.077684    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/026d0a04-126f-4959-94a2-9300876e2bb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026d0a04-126f-4959-94a2-9300876e2bb1-maas-api-xks-ca\") pod \"026d0a04-126f-4959-94a2-9300876e2bb1\" (UID: \"026d0a04-126f-4959-94a2-9300876e2bb1\") "
Nov 02 12:27:34.077964 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.077934    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-maas-api-openshift-ca" pod "60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1" (UID: "60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:34.078067 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.077938    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026d0a04-126f-4959-94a2-9300876e2bb1-maas-api-openshift-ca" pod "026d0a04-126f-4959-94a2-9300876e2bb1" (UID: "026d0a04-126f-4959-94a2-9300876e2bb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:34.078262 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.078237    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/026d0a04-126f-4959-94a2-9300876e2bb1-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:34.078353 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.078270    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:34.079661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.079639    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-kube-api-access-rl8d2" pod "60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1" (UID: "60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1"). InnerVolumeSpecName "kube-api-access-rl8d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:34.079790 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.079774    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-maas-api-xks-ca" pod "60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1" (UID: "60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:34.079857 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.079835    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026d0a04-126f-4959-94a2-9300876e2bb1-kube-api-access-vkxb2" pod "026d0a04-126f-4959-94a2-9300876e2bb1" (UID: "026d0a04-126f-4959-94a2-9300876e2bb1"). InnerVolumeSpecName "kube-api-access-vkxb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:34.079956 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.079942    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/026d0a04-126f-4959-94a2-9300876e2bb1-maas-api-xks-ca" pod "026d0a04-126f-4959-94a2-9300876e2bb1" (UID: "026d0a04-126f-4959-94a2-9300876e2bb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:34.101239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.101216    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4445878c-kqmcm"]
Nov 02 12:27:34.179294 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.179217    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/026d0a04-126f-4959-94a2-9300876e2bb1-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:34.179294 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.179248    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl8d2\" (UniqueName: \"kubernetes.io/projected/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-kube-api-access-rl8d2\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:34.179294 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.179263    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkxb2\" (UniqueName: \"kubernetes.io/projected/026d0a04-126f-4959-94a2-9300876e2bb1-kube-api-access-vkxb2\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:34.179294 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.179276    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:34.840779 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.840747    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"
Nov 02 12:27:34.840966 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.840793    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5" event={"ID":"60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1","Type":"ContainerDied","Data":"bfddf70c10a3a702d24fdc8223a52889f690fab53ca914fa6dafe41d986922f0"}
Nov 02 12:27:34.840966 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.840852    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfddf70c10a3a702d24fdc8223a52889f690fab53ca914fa6dafe41d986922f0"
Nov 02 12:27:34.842345 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.842324    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"
Nov 02 12:27:34.842506 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.842364    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz" event={"ID":"026d0a04-126f-4959-94a2-9300876e2bb1","Type":"ContainerDied","Data":"39824f9c7642403c826bfe6c75651d36087d75d47d0769350925104c8251f7ab"}
Nov 02 12:27:34.842506 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:34.842405    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39824f9c7642403c826bfe6c75651d36087d75d47d0769350925104c8251f7ab"
Nov 02 12:27:35.583395 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:35.583362    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b4233ad-7e42-44ee-b826-6106d6278ee7" path="/var/lib/kubelet/pods/2b4233ad-7e42-44ee-b826-6106d6278ee7/volumes"
Nov 02 12:27:36.757450 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:36.757422    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"
Nov 02 12:27:41.170064 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:41.170031    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:27:42.898150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:42.898110    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"]
Nov 02 12:27:42.899310 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:42.899283    4464 cpu_manager.go:414] "containerMap: removing container" podUID="026d0a04-126f-4959-94a2-9300876e2bb1" containerName="revoke-keys"
Nov 02 12:27:42.899310 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:42.899309    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026d0a04-126f-4959-94a2-9300876e2bb1" containerName="revoke-keys"
Nov 02 12:27:42.899512 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:42.899345    4464 cpu_manager.go:414] "containerMap: removing container" podUID="60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1" containerName="revoke-keys"
Nov 02 12:27:42.899512 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:42.899354    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1" containerName="revoke-keys"
Nov 02 12:27:42.899635 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:42.899548    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="026d0a04-126f-4959-94a2-9300876e2bb1" containerName="revoke-keys"
Nov 02 12:27:42.899635 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:42.899570    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1" containerName="revoke-keys"
Nov 02 12:27:42.903794 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:42.903772    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"
Nov 02 12:27:42.959879 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:42.959827    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"]
Nov 02 12:27:43.004111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:43.004067    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:27:43.004808 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:43.004753    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:27:43.004990 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:43.004775    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:27:43.008520 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:43.008492    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:27:43.051954 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:43.051915    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"]
Nov 02 12:27:43.059378 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:43.059349    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6111b248-cpljf"]
Nov 02 12:27:43.089869 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:43.089827    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3c97f6c-z2nrg\" (UID: \"f7f82fe7-86f1-494b-ba79-dbec14b80cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"
Nov 02 12:27:43.090106 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:43.090084    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3c97f6c-z2nrg\" (UID: \"f7f82fe7-86f1-494b-ba79-dbec14b80cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"
Nov 02 12:27:43.090189 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:43.090120    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-247wc\" (UniqueName: \"kubernetes.io/projected/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-kube-api-access-247wc\") pod \"maas-api-revoke-sub-e3c97f6c-z2nrg\" (UID: \"f7f82fe7-86f1-494b-ba79-dbec14b80cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"
Nov 02 12:27:43.192034 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:43.191928    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3c97f6c-z2nrg\" (UID: \"f7f82fe7-86f1-494b-ba79-dbec14b80cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"
Nov 02 12:27:43.192034 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:43.191993    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-247wc\" (UniqueName: \"kubernetes.io/projected/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-kube-api-access-247wc\") pod \"maas-api-revoke-sub-e3c97f6c-z2nrg\" (UID: \"f7f82fe7-86f1-494b-ba79-dbec14b80cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"
Nov 02 12:27:43.192275 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:43.192108    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3c97f6c-z2nrg\" (UID: \"f7f82fe7-86f1-494b-ba79-dbec14b80cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"
Nov 02 12:27:43.193127 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:43.193098    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3c97f6c-z2nrg\" (UID: \"f7f82fe7-86f1-494b-ba79-dbec14b80cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"
Nov 02 12:27:43.195255 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:43.195232    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3c97f6c-z2nrg\" (UID: \"f7f82fe7-86f1-494b-ba79-dbec14b80cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"
Nov 02 12:27:43.256015 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:43.255697    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-247wc\" (UniqueName: \"kubernetes.io/projected/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-kube-api-access-247wc\") pod \"maas-api-revoke-sub-e3c97f6c-z2nrg\" (UID: \"f7f82fe7-86f1-494b-ba79-dbec14b80cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"
Nov 02 12:27:43.549040 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:43.545149    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"
Nov 02 12:27:43.643031 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:43.631709    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08a4c94e-3e0e-419d-95ed-01659117c396" path="/var/lib/kubelet/pods/08a4c94e-3e0e-419d-95ed-01659117c396/volumes"
Nov 02 12:27:44.710996 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:44.709806    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"]
Nov 02 12:27:45.448891 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:45.448772    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg" event={"ID":"f7f82fe7-86f1-494b-ba79-dbec14b80cc0","Type":"ContainerStarted","Data":"19259695bd0f46253a252230afd5bd1106c82859210318787c35376ad7c917f4"}
Nov 02 12:27:45.463043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:45.463006    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg" event={"ID":"f7f82fe7-86f1-494b-ba79-dbec14b80cc0","Type":"ContainerStarted","Data":"70f6a8f72c7d3b05896e78dba952b8a246072b742bbe1b16f1605f8be85f9ebf"}
Nov 02 12:27:45.758431 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:45.758366    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg" podStartSLOduration=3.758345008 podStartE2EDuration="3.758345008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:45.618362248 +0000 UTC m=+1894.182070669" watchObservedRunningTime="2026-11-02 12:27:45.758345008 +0000 UTC m=+1894.322053418"
Nov 02 12:27:46.462467 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:46.462187    4464 generic.go:356] "Generic (PLEG): container finished" podID="f7f82fe7-86f1-494b-ba79-dbec14b80cc0" containerID="19259695bd0f46253a252230afd5bd1106c82859210318787c35376ad7c917f4" exitCode=0
Nov 02 12:27:46.462467 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:46.462437    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg" event={"ID":"f7f82fe7-86f1-494b-ba79-dbec14b80cc0","Type":"ContainerDied","Data":"19259695bd0f46253a252230afd5bd1106c82859210318787c35376ad7c917f4"}
Nov 02 12:27:47.626195 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:47.626171    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"
Nov 02 12:27:47.741864 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:47.741782    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-kube-api-access-247wc\" (UniqueName: \"kubernetes.io/projected/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-kube-api-access-247wc\") pod \"f7f82fe7-86f1-494b-ba79-dbec14b80cc0\" (UID: \"f7f82fe7-86f1-494b-ba79-dbec14b80cc0\") "
Nov 02 12:27:47.742046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:47.741929    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-maas-api-xks-ca\") pod \"f7f82fe7-86f1-494b-ba79-dbec14b80cc0\" (UID: \"f7f82fe7-86f1-494b-ba79-dbec14b80cc0\") "
Nov 02 12:27:47.742046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:47.742025    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-maas-api-openshift-ca\") pod \"f7f82fe7-86f1-494b-ba79-dbec14b80cc0\" (UID: \"f7f82fe7-86f1-494b-ba79-dbec14b80cc0\") "
Nov 02 12:27:47.742542 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:47.742506    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-maas-api-openshift-ca" pod "f7f82fe7-86f1-494b-ba79-dbec14b80cc0" (UID: "f7f82fe7-86f1-494b-ba79-dbec14b80cc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:47.744044 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:47.744016    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-kube-api-access-247wc" pod "f7f82fe7-86f1-494b-ba79-dbec14b80cc0" (UID: "f7f82fe7-86f1-494b-ba79-dbec14b80cc0"). InnerVolumeSpecName "kube-api-access-247wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:47.744175 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:47.744120    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-maas-api-xks-ca" pod "f7f82fe7-86f1-494b-ba79-dbec14b80cc0" (UID: "f7f82fe7-86f1-494b-ba79-dbec14b80cc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:47.844078 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:47.844045    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:47.844078 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:47.844077    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:47.844263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:47.844094    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-247wc\" (UniqueName: \"kubernetes.io/projected/f7f82fe7-86f1-494b-ba79-dbec14b80cc0-kube-api-access-247wc\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:48.477407 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:48.477379    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"
Nov 02 12:27:48.477593 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:48.477411    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg" event={"ID":"f7f82fe7-86f1-494b-ba79-dbec14b80cc0","Type":"ContainerDied","Data":"70f6a8f72c7d3b05896e78dba952b8a246072b742bbe1b16f1605f8be85f9ebf"}
Nov 02 12:27:48.477593 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:48.477467    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70f6a8f72c7d3b05896e78dba952b8a246072b742bbe1b16f1605f8be85f9ebf"
Nov 02 12:27:53.905742 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:53.905700    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"]
Nov 02 12:27:53.907399 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:53.907373    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f7f82fe7-86f1-494b-ba79-dbec14b80cc0" containerName="revoke-keys"
Nov 02 12:27:53.907555 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:53.907538    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f82fe7-86f1-494b-ba79-dbec14b80cc0" containerName="revoke-keys"
Nov 02 12:27:53.907827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:53.907811    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f82fe7-86f1-494b-ba79-dbec14b80cc0" containerName="revoke-keys"
Nov 02 12:27:53.920199 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:53.920019    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"
Nov 02 12:27:53.972537 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:53.969223    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"]
Nov 02 12:27:54.013509 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:54.013480    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:27:54.013784 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:54.013767    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:27:54.014193 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:54.014174    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:27:54.014415 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:54.014398    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:27:54.039752 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:54.039716    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97kld\" (UniqueName: \"kubernetes.io/projected/97721af5-a292-479f-b254-e989912ebace-kube-api-access-97kld\") pod \"maas-api-revoke-sub-bc4c8313-swfdf\" (UID: \"97721af5-a292-479f-b254-e989912ebace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"
Nov 02 12:27:54.040001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:54.039920    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97721af5-a292-479f-b254-e989912ebace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc4c8313-swfdf\" (UID: \"97721af5-a292-479f-b254-e989912ebace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"
Nov 02 12:27:54.040106 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:54.040035    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97721af5-a292-479f-b254-e989912ebace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc4c8313-swfdf\" (UID: \"97721af5-a292-479f-b254-e989912ebace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"
Nov 02 12:27:54.141685 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:54.141647    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97721af5-a292-479f-b254-e989912ebace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc4c8313-swfdf\" (UID: \"97721af5-a292-479f-b254-e989912ebace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"
Nov 02 12:27:54.141893 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:54.141709    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97721af5-a292-479f-b254-e989912ebace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc4c8313-swfdf\" (UID: \"97721af5-a292-479f-b254-e989912ebace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"
Nov 02 12:27:54.141893 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:54.141825    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97kld\" (UniqueName: \"kubernetes.io/projected/97721af5-a292-479f-b254-e989912ebace-kube-api-access-97kld\") pod \"maas-api-revoke-sub-bc4c8313-swfdf\" (UID: \"97721af5-a292-479f-b254-e989912ebace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"
Nov 02 12:27:54.142887 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:54.142857    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97721af5-a292-479f-b254-e989912ebace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc4c8313-swfdf\" (UID: \"97721af5-a292-479f-b254-e989912ebace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"
Nov 02 12:27:54.144909 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:54.144885    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97721af5-a292-479f-b254-e989912ebace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc4c8313-swfdf\" (UID: \"97721af5-a292-479f-b254-e989912ebace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"
Nov 02 12:27:54.205311 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:54.205241    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97kld\" (UniqueName: \"kubernetes.io/projected/97721af5-a292-479f-b254-e989912ebace-kube-api-access-97kld\") pod \"maas-api-revoke-sub-bc4c8313-swfdf\" (UID: \"97721af5-a292-479f-b254-e989912ebace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"
Nov 02 12:27:54.248035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:54.248000    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"
Nov 02 12:27:54.664953 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:54.664858    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"]
Nov 02 12:27:54.669593 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:27:54.669564    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97721af5_a292_479f_b254_e989912ebace.slice/crio-5c271949221eaf044ef71aef24792b155a917a08b30716080280aa2211a76043 WatchSource:0}: Error finding container 5c271949221eaf044ef71aef24792b155a917a08b30716080280aa2211a76043: Status 404 returned error can't find the container with id 5c271949221eaf044ef71aef24792b155a917a08b30716080280aa2211a76043
Nov 02 12:27:56.028553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:56.028445    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf" event={"ID":"97721af5-a292-479f-b254-e989912ebace","Type":"ContainerStarted","Data":"92dca447239fc280ecbf97f56f00a390a9397abe43c7176708b82a6acbce7eda"}
Nov 02 12:27:56.028553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:56.028510    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf" event={"ID":"97721af5-a292-479f-b254-e989912ebace","Type":"ContainerStarted","Data":"5c271949221eaf044ef71aef24792b155a917a08b30716080280aa2211a76043"}
Nov 02 12:27:56.134997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:56.131823    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf" podStartSLOduration=3.131804384 podStartE2EDuration="3.131804384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:56.131460333 +0000 UTC m=+1904.695168755" watchObservedRunningTime="2026-11-02 12:27:56.131804384 +0000 UTC m=+1904.695512794"
Nov 02 12:27:57.007942 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:57.007905    4464 generic.go:356] "Generic (PLEG): container finished" podID="97721af5-a292-479f-b254-e989912ebace" containerID="92dca447239fc280ecbf97f56f00a390a9397abe43c7176708b82a6acbce7eda" exitCode=0
Nov 02 12:27:57.014320 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:57.013585    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf" event={"ID":"97721af5-a292-479f-b254-e989912ebace","Type":"ContainerDied","Data":"92dca447239fc280ecbf97f56f00a390a9397abe43c7176708b82a6acbce7eda"}
Nov 02 12:27:58.533691 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:27:58.532844    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:27:58.596960 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:58.596658    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"
Nov 02 12:27:58.633369 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:58.633330    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97721af5-a292-479f-b254-e989912ebace-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97721af5-a292-479f-b254-e989912ebace-maas-api-openshift-ca\") pod \"97721af5-a292-479f-b254-e989912ebace\" (UID: \"97721af5-a292-479f-b254-e989912ebace\") "
Nov 02 12:27:58.633556 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:58.633417    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97721af5-a292-479f-b254-e989912ebace-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97721af5-a292-479f-b254-e989912ebace-maas-api-xks-ca\") pod \"97721af5-a292-479f-b254-e989912ebace\" (UID: \"97721af5-a292-479f-b254-e989912ebace\") "
Nov 02 12:27:58.633556 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:58.633509    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97721af5-a292-479f-b254-e989912ebace-kube-api-access-97kld\" (UniqueName: \"kubernetes.io/projected/97721af5-a292-479f-b254-e989912ebace-kube-api-access-97kld\") pod \"97721af5-a292-479f-b254-e989912ebace\" (UID: \"97721af5-a292-479f-b254-e989912ebace\") "
Nov 02 12:27:58.633694 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:58.633669    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97721af5-a292-479f-b254-e989912ebace-maas-api-openshift-ca" pod "97721af5-a292-479f-b254-e989912ebace" (UID: "97721af5-a292-479f-b254-e989912ebace"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:58.634354 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:58.634327    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97721af5-a292-479f-b254-e989912ebace-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:58.637346 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:58.637322    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97721af5-a292-479f-b254-e989912ebace-maas-api-xks-ca" pod "97721af5-a292-479f-b254-e989912ebace" (UID: "97721af5-a292-479f-b254-e989912ebace"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:58.639192 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:58.639165    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97721af5-a292-479f-b254-e989912ebace-kube-api-access-97kld" pod "97721af5-a292-479f-b254-e989912ebace" (UID: "97721af5-a292-479f-b254-e989912ebace"). InnerVolumeSpecName "kube-api-access-97kld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:58.738437 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:58.738400    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97721af5-a292-479f-b254-e989912ebace-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:58.738745 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:58.738716    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97kld\" (UniqueName: \"kubernetes.io/projected/97721af5-a292-479f-b254-e989912ebace-kube-api-access-97kld\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:27:59.079166 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:59.079055    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf" event={"ID":"97721af5-a292-479f-b254-e989912ebace","Type":"ContainerDied","Data":"5c271949221eaf044ef71aef24792b155a917a08b30716080280aa2211a76043"}
Nov 02 12:27:59.079166 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:59.079112    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c271949221eaf044ef71aef24792b155a917a08b30716080280aa2211a76043"
Nov 02 12:27:59.079166 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:27:59.079075    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"
Nov 02 12:28:06.971054 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:06.971019    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:28:13.115848 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:13.115817    4464 scope.go:122] "RemoveContainer" containerID="e5d7c43ae91c1c0938090002bd4e167ef8f88ca461bfe4bca26d9c4f748fd471"
Nov 02 12:28:13.125545 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:13.125525    4464 scope.go:122] "RemoveContainer" containerID="79d417bc1334365c48a2cfc5b2551c7dd688bff1486e47fea6cd5d97dc21f4ad"
Nov 02 12:28:13.134391 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:13.134366    4464 scope.go:122] "RemoveContainer" containerID="facfca7dfa8f10ac50c87b4fd165b3df362b190df8be17f175c337594b2c42f9"
Nov 02 12:28:13.142623 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:13.142606    4464 scope.go:122] "RemoveContainer" containerID="dbf5f4c6f02931a22852ddb4b7898054177c790d5e66ef957732ea562041cd0e"
Nov 02 12:28:13.151022 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:13.151004    4464 scope.go:122] "RemoveContainer" containerID="8f5691584389afe8c5066da584265c0521528cdffbdbc4757aacf9af759f2c36"
Nov 02 12:28:23.050343 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:23.050310    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"]
Nov 02 12:28:23.083488 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:23.083439    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-962c869b-hx6f7"]
Nov 02 12:28:23.582041 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:23.582011    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9dbd750-b043-4950-b08c-ef027ef51ef8" path="/var/lib/kubelet/pods/e9dbd750-b043-4950-b08c-ef027ef51ef8/volumes"
Nov 02 12:28:25.239073 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:25.239001    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:28:28.292877 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:28:28.292841    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:28:43.300703 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.300670    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"]
Nov 02 12:28:43.301363 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.301348    4464 cpu_manager.go:414] "containerMap: removing container" podUID="97721af5-a292-479f-b254-e989912ebace" containerName="revoke-keys"
Nov 02 12:28:43.301410 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.301364    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97721af5-a292-479f-b254-e989912ebace" containerName="revoke-keys"
Nov 02 12:28:43.301509 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.301499    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="97721af5-a292-479f-b254-e989912ebace" containerName="revoke-keys"
Nov 02 12:28:43.304946 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.304931    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"
Nov 02 12:28:43.362072 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.362044    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"]
Nov 02 12:28:43.408585 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.408555    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:28:43.409050 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.409012    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:28:43.409050 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.409027    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:28:43.409050 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.409012    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:28:43.482985 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.482943    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49da3a69-3805-4aed-8337-5b818bf8280b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh\" (UID: \"49da3a69-3805-4aed-8337-5b818bf8280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"
Nov 02 12:28:43.483157 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.483090    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmwnr\" (UniqueName: \"kubernetes.io/projected/49da3a69-3805-4aed-8337-5b818bf8280b-kube-api-access-jmwnr\") pod \"maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh\" (UID: \"49da3a69-3805-4aed-8337-5b818bf8280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"
Nov 02 12:28:43.483157 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.483140    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49da3a69-3805-4aed-8337-5b818bf8280b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh\" (UID: \"49da3a69-3805-4aed-8337-5b818bf8280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"
Nov 02 12:28:43.584906 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.584821    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmwnr\" (UniqueName: \"kubernetes.io/projected/49da3a69-3805-4aed-8337-5b818bf8280b-kube-api-access-jmwnr\") pod \"maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh\" (UID: \"49da3a69-3805-4aed-8337-5b818bf8280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"
Nov 02 12:28:43.584906 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.584884    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49da3a69-3805-4aed-8337-5b818bf8280b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh\" (UID: \"49da3a69-3805-4aed-8337-5b818bf8280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"
Nov 02 12:28:43.585170 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.585030    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49da3a69-3805-4aed-8337-5b818bf8280b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh\" (UID: \"49da3a69-3805-4aed-8337-5b818bf8280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"
Nov 02 12:28:43.585531 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.585500    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49da3a69-3805-4aed-8337-5b818bf8280b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh\" (UID: \"49da3a69-3805-4aed-8337-5b818bf8280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"
Nov 02 12:28:43.587245 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.587224    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49da3a69-3805-4aed-8337-5b818bf8280b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh\" (UID: \"49da3a69-3805-4aed-8337-5b818bf8280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"
Nov 02 12:28:43.649483 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.649457    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmwnr\" (UniqueName: \"kubernetes.io/projected/49da3a69-3805-4aed-8337-5b818bf8280b-kube-api-access-jmwnr\") pod \"maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh\" (UID: \"49da3a69-3805-4aed-8337-5b818bf8280b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"
Nov 02 12:28:43.919515 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:43.919427    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"
Nov 02 12:28:44.297474 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:44.297440    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"]
Nov 02 12:28:44.297619 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:28:44.297504    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49da3a69_3805_4aed_8337_5b818bf8280b.slice/crio-67f55875c2b13a577529ae2abc01c160fd4fc6b171da9100152d1074c28a519e WatchSource:0}: Error finding container 67f55875c2b13a577529ae2abc01c160fd4fc6b171da9100152d1074c28a519e: Status 404 returned error can't find the container with id 67f55875c2b13a577529ae2abc01c160fd4fc6b171da9100152d1074c28a519e
Nov 02 12:28:44.413121 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:44.413084    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh" event={"ID":"49da3a69-3805-4aed-8337-5b818bf8280b","Type":"ContainerStarted","Data":"67f55875c2b13a577529ae2abc01c160fd4fc6b171da9100152d1074c28a519e"}
Nov 02 12:28:45.422253 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:45.422168    4464 generic.go:356] "Generic (PLEG): container finished" podID="49da3a69-3805-4aed-8337-5b818bf8280b" containerID="e7e74b79b565a83946ded02528077c5070dda02980499d758599e6237232a97c" exitCode=0
Nov 02 12:28:45.422651 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:45.422259    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh" event={"ID":"49da3a69-3805-4aed-8337-5b818bf8280b","Type":"ContainerDied","Data":"e7e74b79b565a83946ded02528077c5070dda02980499d758599e6237232a97c"}
Nov 02 12:28:46.584646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:46.584619    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"
Nov 02 12:28:46.719439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:46.719382    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49da3a69-3805-4aed-8337-5b818bf8280b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49da3a69-3805-4aed-8337-5b818bf8280b-maas-api-openshift-ca\") pod \"49da3a69-3805-4aed-8337-5b818bf8280b\" (UID: \"49da3a69-3805-4aed-8337-5b818bf8280b\") "
Nov 02 12:28:46.719626 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:46.719490    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49da3a69-3805-4aed-8337-5b818bf8280b-kube-api-access-jmwnr\" (UniqueName: \"kubernetes.io/projected/49da3a69-3805-4aed-8337-5b818bf8280b-kube-api-access-jmwnr\") pod \"49da3a69-3805-4aed-8337-5b818bf8280b\" (UID: \"49da3a69-3805-4aed-8337-5b818bf8280b\") "
Nov 02 12:28:46.719626 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:46.719529    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49da3a69-3805-4aed-8337-5b818bf8280b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49da3a69-3805-4aed-8337-5b818bf8280b-maas-api-xks-ca\") pod \"49da3a69-3805-4aed-8337-5b818bf8280b\" (UID: \"49da3a69-3805-4aed-8337-5b818bf8280b\") "
Nov 02 12:28:46.719834 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:46.719806    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49da3a69-3805-4aed-8337-5b818bf8280b-maas-api-openshift-ca" pod "49da3a69-3805-4aed-8337-5b818bf8280b" (UID: "49da3a69-3805-4aed-8337-5b818bf8280b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:28:46.720110 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:46.720087    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49da3a69-3805-4aed-8337-5b818bf8280b-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:28:46.721784 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:46.721757    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49da3a69-3805-4aed-8337-5b818bf8280b-maas-api-xks-ca" pod "49da3a69-3805-4aed-8337-5b818bf8280b" (UID: "49da3a69-3805-4aed-8337-5b818bf8280b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:28:46.722148 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:46.722125    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49da3a69-3805-4aed-8337-5b818bf8280b-kube-api-access-jmwnr" pod "49da3a69-3805-4aed-8337-5b818bf8280b" (UID: "49da3a69-3805-4aed-8337-5b818bf8280b"). InnerVolumeSpecName "kube-api-access-jmwnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:28:46.821187 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:46.821151    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmwnr\" (UniqueName: \"kubernetes.io/projected/49da3a69-3805-4aed-8337-5b818bf8280b-kube-api-access-jmwnr\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:28:46.821187 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:46.821182    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49da3a69-3805-4aed-8337-5b818bf8280b-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:28:47.438202 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:47.438168    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"
Nov 02 12:28:47.438381 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:47.438206    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh" event={"ID":"49da3a69-3805-4aed-8337-5b818bf8280b","Type":"ContainerDied","Data":"67f55875c2b13a577529ae2abc01c160fd4fc6b171da9100152d1074c28a519e"}
Nov 02 12:28:47.438381 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:47.438256    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67f55875c2b13a577529ae2abc01c160fd4fc6b171da9100152d1074c28a519e"
Nov 02 12:28:54.974679 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:54.974638    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"]
Nov 02 12:28:58.292438 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:28:58.292401    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:28:59.975682 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:28:59.975640    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7" podUID="3aa0c066-d198-4b91-bee6-9859d9b0de9c" containerName="payload-pre-processing" containerID="cri-o://96081459698f89baee16bde4f047c108adaff0cbc88963396ab935575908c743" gracePeriod=25
Nov 02 12:29:00.232028 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:00.231950    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"
Nov 02 12:29:00.313174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:00.312468    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa0c066-d198-4b91-bee6-9859d9b0de9c-kube-api-access-jqjqt\" (UniqueName: \"kubernetes.io/projected/3aa0c066-d198-4b91-bee6-9859d9b0de9c-kube-api-access-jqjqt\") pod \"3aa0c066-d198-4b91-bee6-9859d9b0de9c\" (UID: \"3aa0c066-d198-4b91-bee6-9859d9b0de9c\") "
Nov 02 12:29:00.313174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:00.312590    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa0c066-d198-4b91-bee6-9859d9b0de9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa0c066-d198-4b91-bee6-9859d9b0de9c-plugins-config-volume\") pod \"3aa0c066-d198-4b91-bee6-9859d9b0de9c\" (UID: \"3aa0c066-d198-4b91-bee6-9859d9b0de9c\") "
Nov 02 12:29:00.313401 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:00.313266    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa0c066-d198-4b91-bee6-9859d9b0de9c-plugins-config-volume" pod "3aa0c066-d198-4b91-bee6-9859d9b0de9c" (UID: "3aa0c066-d198-4b91-bee6-9859d9b0de9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:29:00.315280 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:00.315255    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa0c066-d198-4b91-bee6-9859d9b0de9c-kube-api-access-jqjqt" pod "3aa0c066-d198-4b91-bee6-9859d9b0de9c" (UID: "3aa0c066-d198-4b91-bee6-9859d9b0de9c"). InnerVolumeSpecName "kube-api-access-jqjqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:29:00.413892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:00.413860    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqjqt\" (UniqueName: \"kubernetes.io/projected/3aa0c066-d198-4b91-bee6-9859d9b0de9c-kube-api-access-jqjqt\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:00.413892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:00.413887    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aa0c066-d198-4b91-bee6-9859d9b0de9c-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:00.610218 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:00.610136    4464 generic.go:356] "Generic (PLEG): container finished" podID="3aa0c066-d198-4b91-bee6-9859d9b0de9c" containerID="96081459698f89baee16bde4f047c108adaff0cbc88963396ab935575908c743" exitCode=0
Nov 02 12:29:00.610218 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:00.610178    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"
Nov 02 12:29:00.610398 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:00.610223    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7" event={"ID":"3aa0c066-d198-4b91-bee6-9859d9b0de9c","Type":"ContainerDied","Data":"96081459698f89baee16bde4f047c108adaff0cbc88963396ab935575908c743"}
Nov 02 12:29:00.610398 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:00.610268    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7" event={"ID":"3aa0c066-d198-4b91-bee6-9859d9b0de9c","Type":"ContainerDied","Data":"ecc7a1be6d7bccb855ac1269e853e76311f30074e5804e1c577612c8dd7de6ac"}
Nov 02 12:29:00.610398 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:00.610288    4464 scope.go:122] "RemoveContainer" containerID="96081459698f89baee16bde4f047c108adaff0cbc88963396ab935575908c743"
Nov 02 12:29:00.620609 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:00.620592    4464 scope.go:122] "RemoveContainer" containerID="96081459698f89baee16bde4f047c108adaff0cbc88963396ab935575908c743"
Nov 02 12:29:00.620890 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:29:00.620866    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96081459698f89baee16bde4f047c108adaff0cbc88963396ab935575908c743\": container with ID starting with 96081459698f89baee16bde4f047c108adaff0cbc88963396ab935575908c743 not found: ID does not exist" containerID="96081459698f89baee16bde4f047c108adaff0cbc88963396ab935575908c743"
Nov 02 12:29:00.620949 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:00.620897    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96081459698f89baee16bde4f047c108adaff0cbc88963396ab935575908c743"} err="failed to get container status \"96081459698f89baee16bde4f047c108adaff0cbc88963396ab935575908c743\": rpc error: code = NotFound desc = could not find container \"96081459698f89baee16bde4f047c108adaff0cbc88963396ab935575908c743\": container with ID starting with 96081459698f89baee16bde4f047c108adaff0cbc88963396ab935575908c743 not found: ID does not exist"
Nov 02 12:29:00.778307 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:00.778277    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"]
Nov 02 12:29:00.784297 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:00.784262    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ccee868d-566b985687-vxcc7"]
Nov 02 12:29:01.589638 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:01.589608    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa0c066-d198-4b91-bee6-9859d9b0de9c" path="/var/lib/kubelet/pods/3aa0c066-d198-4b91-bee6-9859d9b0de9c/volumes"
Nov 02 12:29:12.027832 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.027798    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"]
Nov 02 12:29:12.028439 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.028423    4464 cpu_manager.go:414] "containerMap: removing container" podUID="49da3a69-3805-4aed-8337-5b818bf8280b" containerName="revoke-keys"
Nov 02 12:29:12.028483 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.028440    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49da3a69-3805-4aed-8337-5b818bf8280b" containerName="revoke-keys"
Nov 02 12:29:12.028483 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.028466    4464 cpu_manager.go:414] "containerMap: removing container" podUID="3aa0c066-d198-4b91-bee6-9859d9b0de9c" containerName="payload-pre-processing"
Nov 02 12:29:12.028483 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.028471    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa0c066-d198-4b91-bee6-9859d9b0de9c" containerName="payload-pre-processing"
Nov 02 12:29:12.028580 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.028563    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa0c066-d198-4b91-bee6-9859d9b0de9c" containerName="payload-pre-processing"
Nov 02 12:29:12.028614 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.028585    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="49da3a69-3805-4aed-8337-5b818bf8280b" containerName="revoke-keys"
Nov 02 12:29:12.031889 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.031871    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"
Nov 02 12:29:12.067897 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.067871    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d81c8b-sbx6n\" (UID: \"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"
Nov 02 12:29:12.068073 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.068054    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8vxr\" (UniqueName: \"kubernetes.io/projected/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-kube-api-access-d8vxr\") pod \"maas-api-revoke-sub-a3d81c8b-sbx6n\" (UID: \"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"
Nov 02 12:29:12.068201 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.068182    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d81c8b-sbx6n\" (UID: \"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"
Nov 02 12:29:12.135470 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.135433    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:29:12.135470 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.135434    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:29:12.135470 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.135472    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:29:12.135762 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.135501    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:29:12.138945 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.138916    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"]
Nov 02 12:29:12.169315 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.169290    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8vxr\" (UniqueName: \"kubernetes.io/projected/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-kube-api-access-d8vxr\") pod \"maas-api-revoke-sub-a3d81c8b-sbx6n\" (UID: \"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"
Nov 02 12:29:12.169434 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.169404    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d81c8b-sbx6n\" (UID: \"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"
Nov 02 12:29:12.169511 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.169495    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d81c8b-sbx6n\" (UID: \"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"
Nov 02 12:29:12.170136 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.170118    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3d81c8b-sbx6n\" (UID: \"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"
Nov 02 12:29:12.171740 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.171720    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3d81c8b-sbx6n\" (UID: \"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"
Nov 02 12:29:12.232116 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.232075    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8vxr\" (UniqueName: \"kubernetes.io/projected/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-kube-api-access-d8vxr\") pod \"maas-api-revoke-sub-a3d81c8b-sbx6n\" (UID: \"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"
Nov 02 12:29:12.345484 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.345406    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"
Nov 02 12:29:12.746714 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:12.746684    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"]
Nov 02 12:29:12.746835 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:29:12.746688    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d7c4c4_550a_4d5d_a16e_c0a554bce3f3.slice/crio-8538dbfb4dcbbd993107a1c9a999bcf01a4f3c4ba2df954359db4096c1c14710 WatchSource:0}: Error finding container 8538dbfb4dcbbd993107a1c9a999bcf01a4f3c4ba2df954359db4096c1c14710: Status 404 returned error can't find the container with id 8538dbfb4dcbbd993107a1c9a999bcf01a4f3c4ba2df954359db4096c1c14710
Nov 02 12:29:13.209431 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:13.209398    4464 scope.go:122] "RemoveContainer" containerID="9c3c46e8080118aff1e36c5af62128e9386474f6a40440bd3f7c382e9739f72f"
Nov 02 12:29:13.219273 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:13.219251    4464 scope.go:122] "RemoveContainer" containerID="6cc2c2f06ee9ea6d364bb2cb7de3a4c5f0d3c25b228c386382b96837cce600af"
Nov 02 12:29:13.228947 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:13.228925    4464 scope.go:122] "RemoveContainer" containerID="f8984e9965b540c6c70f85b4aa70f89f10cd9c654215df9e0e7e6ae43b30c0cf"
Nov 02 12:29:13.237646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:13.237627    4464 scope.go:122] "RemoveContainer" containerID="2b23e24ae17c5399f4cad4facf9bed5b34119f87a45a01d25b2a69d6618165de"
Nov 02 12:29:13.246291 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:13.246269    4464 scope.go:122] "RemoveContainer" containerID="7c5f350dfce1d97037251ad796e37a2aa886a46d5b37385716cf5d7937e7bad7"
Nov 02 12:29:13.256502 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:13.256481    4464 scope.go:122] "RemoveContainer" containerID="219c36379084d5b6521dd43fdbdadf7c81fad38f9eebbac2a473f6a5bcd53acd"
Nov 02 12:29:13.697615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:13.697585    4464 generic.go:356] "Generic (PLEG): container finished" podID="66d7c4c4-550a-4d5d-a16e-c0a554bce3f3" containerID="37a9018dcb72abdff8696bee036d1fe206f74d772a6e4d722e2ff5061eea8fa5" exitCode=0
Nov 02 12:29:13.697817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:13.697675    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n" event={"ID":"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3","Type":"ContainerDied","Data":"37a9018dcb72abdff8696bee036d1fe206f74d772a6e4d722e2ff5061eea8fa5"}
Nov 02 12:29:13.697817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:13.697734    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n" event={"ID":"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3","Type":"ContainerStarted","Data":"8538dbfb4dcbbd993107a1c9a999bcf01a4f3c4ba2df954359db4096c1c14710"}
Nov 02 12:29:14.865732 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:14.865707    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"
Nov 02 12:29:14.915799 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:14.915773    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-maas-api-xks-ca\") pod \"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3\" (UID: \"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3\") "
Nov 02 12:29:14.915916 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:14.915894    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-kube-api-access-d8vxr\" (UniqueName: \"kubernetes.io/projected/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-kube-api-access-d8vxr\") pod \"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3\" (UID: \"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3\") "
Nov 02 12:29:14.916008 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:14.915996    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-maas-api-openshift-ca\") pod \"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3\" (UID: \"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3\") "
Nov 02 12:29:14.916344 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:14.916315    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-maas-api-openshift-ca" pod "66d7c4c4-550a-4d5d-a16e-c0a554bce3f3" (UID: "66d7c4c4-550a-4d5d-a16e-c0a554bce3f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:29:14.917738 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:14.917703    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-maas-api-xks-ca" pod "66d7c4c4-550a-4d5d-a16e-c0a554bce3f3" (UID: "66d7c4c4-550a-4d5d-a16e-c0a554bce3f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:29:14.917827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:14.917807    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-kube-api-access-d8vxr" pod "66d7c4c4-550a-4d5d-a16e-c0a554bce3f3" (UID: "66d7c4c4-550a-4d5d-a16e-c0a554bce3f3"). InnerVolumeSpecName "kube-api-access-d8vxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:29:15.018330 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:15.018293    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:15.018493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:15.018335    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8vxr\" (UniqueName: \"kubernetes.io/projected/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-kube-api-access-d8vxr\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:15.018493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:15.018353    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:15.716026 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:15.715996    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"
Nov 02 12:29:15.716026 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:15.716007    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n" event={"ID":"66d7c4c4-550a-4d5d-a16e-c0a554bce3f3","Type":"ContainerDied","Data":"8538dbfb4dcbbd993107a1c9a999bcf01a4f3c4ba2df954359db4096c1c14710"}
Nov 02 12:29:15.716230 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:15.716065    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8538dbfb4dcbbd993107a1c9a999bcf01a4f3c4ba2df954359db4096c1c14710"
Nov 02 12:29:16.425438 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:16.425405    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"]
Nov 02 12:29:16.475723 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:16.475689    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ccee868d-8zdqh"]
Nov 02 12:29:17.201005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:17.187256    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"]
Nov 02 12:29:17.201005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:17.187504    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8" podUID="052217bc-c40c-4150-8769-32cc113feab6" containerName="istio-proxy" containerID="cri-o://f2f2f9eaabd06e99776bbb21ad5277b6e9fb130f713939378dd387a946f5a7c9" gracePeriod=30
Nov 02 12:29:17.582680 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:17.582647    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49da3a69-3805-4aed-8337-5b818bf8280b" path="/var/lib/kubelet/pods/49da3a69-3805-4aed-8337-5b818bf8280b/volumes"
Nov 02 12:29:22.228336 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.228297    4464 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": read tcp 10.133.0.2:51614->10.133.0.62:15021: read: connection reset by peer" start-of-body=
Nov 02 12:29:22.228723 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.228377    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8" podUID="052217bc-c40c-4150-8769-32cc113feab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": read tcp 10.133.0.2:51614->10.133.0.62:15021: read: connection reset by peer"
Nov 02 12:29:22.433236 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.433213    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:29:22.561794 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.561721    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/052217bc-c40c-4150-8769-32cc113feab6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052217bc-c40c-4150-8769-32cc113feab6-istiod-ca-cert\") pod \"052217bc-c40c-4150-8769-32cc113feab6\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") "
Nov 02 12:29:22.561794 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.561769    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-istio-data\") pod \"052217bc-c40c-4150-8769-32cc113feab6\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") "
Nov 02 12:29:22.561794 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.561789    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052217bc-c40c-4150-8769-32cc113feab6-kube-api-access-qt6kg\" (UniqueName: \"kubernetes.io/projected/052217bc-c40c-4150-8769-32cc113feab6-kube-api-access-qt6kg\") pod \"052217bc-c40c-4150-8769-32cc113feab6\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") "
Nov 02 12:29:22.562135 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.561874    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-workload-certs\") pod \"052217bc-c40c-4150-8769-32cc113feab6\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") "
Nov 02 12:29:22.562135 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.561932    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-istio-envoy\") pod \"052217bc-c40c-4150-8769-32cc113feab6\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") "
Nov 02 12:29:22.562135 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.562040    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-credential-socket\") pod \"052217bc-c40c-4150-8769-32cc113feab6\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") "
Nov 02 12:29:22.562135 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.562094    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052217bc-c40c-4150-8769-32cc113feab6-istio-token\" (UniqueName: \"kubernetes.io/projected/052217bc-c40c-4150-8769-32cc113feab6-istio-token\") pod \"052217bc-c40c-4150-8769-32cc113feab6\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") "
Nov 02 12:29:22.562135 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.562108    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-istio-data" pod "052217bc-c40c-4150-8769-32cc113feab6" (UID: "052217bc-c40c-4150-8769-32cc113feab6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:22.562402 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.562154    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-workload-socket\") pod \"052217bc-c40c-4150-8769-32cc113feab6\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") "
Nov 02 12:29:22.562402 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.562200    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/052217bc-c40c-4150-8769-32cc113feab6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052217bc-c40c-4150-8769-32cc113feab6-istio-podinfo\") pod \"052217bc-c40c-4150-8769-32cc113feab6\" (UID: \"052217bc-c40c-4150-8769-32cc113feab6\") "
Nov 02 12:29:22.562402 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.562220    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-workload-certs" pod "052217bc-c40c-4150-8769-32cc113feab6" (UID: "052217bc-c40c-4150-8769-32cc113feab6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:22.562402 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.562141    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052217bc-c40c-4150-8769-32cc113feab6-istiod-ca-cert" pod "052217bc-c40c-4150-8769-32cc113feab6" (UID: "052217bc-c40c-4150-8769-32cc113feab6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:29:22.562402 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.562324    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-credential-socket" pod "052217bc-c40c-4150-8769-32cc113feab6" (UID: "052217bc-c40c-4150-8769-32cc113feab6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:22.562631 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.562414    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-workload-socket" pod "052217bc-c40c-4150-8769-32cc113feab6" (UID: "052217bc-c40c-4150-8769-32cc113feab6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:22.562811 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.562788    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:22.562879 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.562818    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:22.562879 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.562835    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:22.562879 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.562850    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052217bc-c40c-4150-8769-32cc113feab6-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:22.562879 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.562865    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:22.564248 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.564225    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052217bc-c40c-4150-8769-32cc113feab6-istio-token" pod "052217bc-c40c-4150-8769-32cc113feab6" (UID: "052217bc-c40c-4150-8769-32cc113feab6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:29:22.564365 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.564225    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-istio-envoy" pod "052217bc-c40c-4150-8769-32cc113feab6" (UID: "052217bc-c40c-4150-8769-32cc113feab6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:22.564653 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.564627    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/052217bc-c40c-4150-8769-32cc113feab6-istio-podinfo" pod "052217bc-c40c-4150-8769-32cc113feab6" (UID: "052217bc-c40c-4150-8769-32cc113feab6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:29:22.564653 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.564638    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052217bc-c40c-4150-8769-32cc113feab6-kube-api-access-qt6kg" pod "052217bc-c40c-4150-8769-32cc113feab6" (UID: "052217bc-c40c-4150-8769-32cc113feab6"). InnerVolumeSpecName "kube-api-access-qt6kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:29:22.664276 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.664244    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052217bc-c40c-4150-8769-32cc113feab6-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:22.664276 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.664278    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052217bc-c40c-4150-8769-32cc113feab6-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:22.664483 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.664295    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052217bc-c40c-4150-8769-32cc113feab6-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:22.664483 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.664311    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt6kg\" (UniqueName: \"kubernetes.io/projected/052217bc-c40c-4150-8769-32cc113feab6-kube-api-access-qt6kg\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:22.762287 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.762247    4464 generic.go:356] "Generic (PLEG): container finished" podID="052217bc-c40c-4150-8769-32cc113feab6" containerID="f2f2f9eaabd06e99776bbb21ad5277b6e9fb130f713939378dd387a946f5a7c9" exitCode=0
Nov 02 12:29:22.762447 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.762292    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"
Nov 02 12:29:22.762447 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.762316    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8" event={"ID":"052217bc-c40c-4150-8769-32cc113feab6","Type":"ContainerDied","Data":"f2f2f9eaabd06e99776bbb21ad5277b6e9fb130f713939378dd387a946f5a7c9"}
Nov 02 12:29:22.762447 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.762363    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8" event={"ID":"052217bc-c40c-4150-8769-32cc113feab6","Type":"ContainerDied","Data":"55c9a6a868d1b46eb72278d4ea46b45309dea5122d1149a25a2712cf2537b8e9"}
Nov 02 12:29:22.762447 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.762388    4464 scope.go:122] "RemoveContainer" containerID="f2f2f9eaabd06e99776bbb21ad5277b6e9fb130f713939378dd387a946f5a7c9"
Nov 02 12:29:22.773204 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.773184    4464 scope.go:122] "RemoveContainer" containerID="f2f2f9eaabd06e99776bbb21ad5277b6e9fb130f713939378dd387a946f5a7c9"
Nov 02 12:29:22.773512 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:29:22.773492    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f2f9eaabd06e99776bbb21ad5277b6e9fb130f713939378dd387a946f5a7c9\": container with ID starting with f2f2f9eaabd06e99776bbb21ad5277b6e9fb130f713939378dd387a946f5a7c9 not found: ID does not exist" containerID="f2f2f9eaabd06e99776bbb21ad5277b6e9fb130f713939378dd387a946f5a7c9"
Nov 02 12:29:22.773607 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.773518    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f2f9eaabd06e99776bbb21ad5277b6e9fb130f713939378dd387a946f5a7c9"} err="failed to get container status \"f2f2f9eaabd06e99776bbb21ad5277b6e9fb130f713939378dd387a946f5a7c9\": rpc error: code = NotFound desc = could not find container \"f2f2f9eaabd06e99776bbb21ad5277b6e9fb130f713939378dd387a946f5a7c9\": container with ID starting with f2f2f9eaabd06e99776bbb21ad5277b6e9fb130f713939378dd387a946f5a7c9 not found: ID does not exist"
Nov 02 12:29:22.931232 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.931043    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"]
Nov 02 12:29:22.934520 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:22.934497    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-qx4f8"]
Nov 02 12:29:23.582509 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:23.582447    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="052217bc-c40c-4150-8769-32cc113feab6" path="/var/lib/kubelet/pods/052217bc-c40c-4150-8769-32cc113feab6/volumes"
Nov 02 12:29:25.282856 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.282810    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"]
Nov 02 12:29:25.283599 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.283570    4464 cpu_manager.go:414] "containerMap: removing container" podUID="66d7c4c4-550a-4d5d-a16e-c0a554bce3f3" containerName="revoke-keys"
Nov 02 12:29:25.283599 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.283591    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d7c4c4-550a-4d5d-a16e-c0a554bce3f3" containerName="revoke-keys"
Nov 02 12:29:25.283732 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.283632    4464 cpu_manager.go:414] "containerMap: removing container" podUID="052217bc-c40c-4150-8769-32cc113feab6" containerName="istio-proxy"
Nov 02 12:29:25.283732 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.283638    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052217bc-c40c-4150-8769-32cc113feab6" containerName="istio-proxy"
Nov 02 12:29:25.283807 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.283757    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="052217bc-c40c-4150-8769-32cc113feab6" containerName="istio-proxy"
Nov 02 12:29:25.283807 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.283774    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d7c4c4-550a-4d5d-a16e-c0a554bce3f3" containerName="revoke-keys"
Nov 02 12:29:25.306631 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.306598    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"
Nov 02 12:29:25.370719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.370689    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"]
Nov 02 12:29:25.441000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.417957    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e079e8-6acf-41b9-b8a8-1a066acccb93-plugins-config-volume\") pod \"payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r\" (UID: \"c1e079e8-6acf-41b9-b8a8-1a066acccb93\") " pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"
Nov 02 12:29:25.441386 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.441352    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qch9\" (UniqueName: \"kubernetes.io/projected/c1e079e8-6acf-41b9-b8a8-1a066acccb93-kube-api-access-6qch9\") pod \"payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r\" (UID: \"c1e079e8-6acf-41b9-b8a8-1a066acccb93\") " pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"
Nov 02 12:29:25.484361 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.484333    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-02304760\""
Nov 02 12:29:25.484779 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.484740    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-02304760-dockercfg-xqxqg\""
Nov 02 12:29:25.493510 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.493482    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"]
Nov 02 12:29:25.498898 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.498877    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"
Nov 02 12:29:25.522301 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.522275    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"]
Nov 02 12:29:25.543453 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.543370    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e079e8-6acf-41b9-b8a8-1a066acccb93-plugins-config-volume\") pod \"payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r\" (UID: \"c1e079e8-6acf-41b9-b8a8-1a066acccb93\") " pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"
Nov 02 12:29:25.543453 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.543434    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qch9\" (UniqueName: \"kubernetes.io/projected/c1e079e8-6acf-41b9-b8a8-1a066acccb93-kube-api-access-6qch9\") pod \"payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r\" (UID: \"c1e079e8-6acf-41b9-b8a8-1a066acccb93\") " pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"
Nov 02 12:29:25.544155 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.544131    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e079e8-6acf-41b9-b8a8-1a066acccb93-plugins-config-volume\") pod \"payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r\" (UID: \"c1e079e8-6acf-41b9-b8a8-1a066acccb93\") " pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"
Nov 02 12:29:25.605602 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.605569    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qch9\" (UniqueName: \"kubernetes.io/projected/c1e079e8-6acf-41b9-b8a8-1a066acccb93-kube-api-access-6qch9\") pod \"payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r\" (UID: \"c1e079e8-6acf-41b9-b8a8-1a066acccb93\") " pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"
Nov 02 12:29:25.644669 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.644625    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89ec611-1f60-49e5-80b9-ef7720ecb732-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t\" (UID: \"d89ec611-1f60-49e5-80b9-ef7720ecb732\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"
Nov 02 12:29:25.644963 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.644943    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-289gg\" (UniqueName: \"kubernetes.io/projected/d89ec611-1f60-49e5-80b9-ef7720ecb732-kube-api-access-289gg\") pod \"payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t\" (UID: \"d89ec611-1f60-49e5-80b9-ef7720ecb732\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"
Nov 02 12:29:25.708512 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.708476    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"
Nov 02 12:29:25.746432 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.746402    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-289gg\" (UniqueName: \"kubernetes.io/projected/d89ec611-1f60-49e5-80b9-ef7720ecb732-kube-api-access-289gg\") pod \"payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t\" (UID: \"d89ec611-1f60-49e5-80b9-ef7720ecb732\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"
Nov 02 12:29:25.746580 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.746519    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89ec611-1f60-49e5-80b9-ef7720ecb732-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t\" (UID: \"d89ec611-1f60-49e5-80b9-ef7720ecb732\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"
Nov 02 12:29:25.747192 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.747168    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89ec611-1f60-49e5-80b9-ef7720ecb732-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t\" (UID: \"d89ec611-1f60-49e5-80b9-ef7720ecb732\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"
Nov 02 12:29:25.809037 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.808882    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-289gg\" (UniqueName: \"kubernetes.io/projected/d89ec611-1f60-49e5-80b9-ef7720ecb732-kube-api-access-289gg\") pod \"payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t\" (UID: \"d89ec611-1f60-49e5-80b9-ef7720ecb732\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"
Nov 02 12:29:25.814432 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:25.814402    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"
Nov 02 12:29:26.088727 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:26.088694    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"]
Nov 02 12:29:26.088840 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:29:26.088826    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e079e8_6acf_41b9_b8a8_1a066acccb93.slice/crio-d09f1ae389f16a8334267c06c3733e0859b2b45810cc9d770b84300690b79f5b WatchSource:0}: Error finding container d09f1ae389f16a8334267c06c3733e0859b2b45810cc9d770b84300690b79f5b: Status 404 returned error can't find the container with id d09f1ae389f16a8334267c06c3733e0859b2b45810cc9d770b84300690b79f5b
Nov 02 12:29:26.196137 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:26.196105    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"]
Nov 02 12:29:26.196641 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:29:26.196613    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89ec611_1f60_49e5_80b9_ef7720ecb732.slice/crio-8eba6f77dd460ac57d696d8e36d6a9c20470fb38740774f4e16175114ac4f816 WatchSource:0}: Error finding container 8eba6f77dd460ac57d696d8e36d6a9c20470fb38740774f4e16175114ac4f816: Status 404 returned error can't find the container with id 8eba6f77dd460ac57d696d8e36d6a9c20470fb38740774f4e16175114ac4f816
Nov 02 12:29:26.382120 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:26.382035    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:29:26.796920 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:26.796883    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t" event={"ID":"d89ec611-1f60-49e5-80b9-ef7720ecb732","Type":"ContainerStarted","Data":"72d49a39903200e08304e614d95f74127f546775c905bab0f0bf49708a30e0a4"}
Nov 02 12:29:26.797234 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:26.796948    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t" event={"ID":"d89ec611-1f60-49e5-80b9-ef7720ecb732","Type":"ContainerStarted","Data":"8eba6f77dd460ac57d696d8e36d6a9c20470fb38740774f4e16175114ac4f816"}
Nov 02 12:29:26.797234 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:26.797147    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"
Nov 02 12:29:26.798769 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:26.798743    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r" event={"ID":"c1e079e8-6acf-41b9-b8a8-1a066acccb93","Type":"ContainerStarted","Data":"22457be605d51c4e0997d215de5f6347b140051197777dfabfd16241ef87ce6f"}
Nov 02 12:29:26.798888 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:26.798784    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r" event={"ID":"c1e079e8-6acf-41b9-b8a8-1a066acccb93","Type":"ContainerStarted","Data":"d09f1ae389f16a8334267c06c3733e0859b2b45810cc9d770b84300690b79f5b"}
Nov 02 12:29:26.798888 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:26.798812    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"
Nov 02 12:29:27.060104 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:27.059962    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t" podStartSLOduration=2.059940839 podStartE2EDuration="2.059940839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:29:26.939359144 +0000 UTC m=+1995.503067555" watchObservedRunningTime="2026-11-02 12:29:27.059940839 +0000 UTC m=+1995.623649249"
Nov 02 12:29:27.061339 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:27.061285    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r" podStartSLOduration=2.061273063 podStartE2EDuration="2.061273063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:29:27.058809208 +0000 UTC m=+1995.622517617" watchObservedRunningTime="2026-11-02 12:29:27.061273063 +0000 UTC m=+1995.624981474"
Nov 02 12:29:28.295115 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:29:28.295079    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:29:34.794364 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:34.794331    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"]
Nov 02 12:29:34.794867 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:34.794599    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7" podUID="e418db77-2841-458b-be0e-e9c69ee81ac1" containerName="istio-proxy" containerID="cri-o://f9000f5976de400442688fe5b64c5b72fb62093139dacc38abeba370b129c778" gracePeriod=30
Nov 02 12:29:34.815775 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:34.815749    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"]
Nov 02 12:29:34.815948 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:34.815927    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7" podUID="dfb8c3c9-4f23-4f81-806a-4d89eab16c48" containerName="istio-proxy" containerID="cri-o://ce6c1613298834fd7d77d28dbe0031064629b693c74cfbbe2956fc4dd171acbd" gracePeriod=30
Nov 02 12:29:36.327427 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:36.327393    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:29:36.327820 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:36.327449    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7" podUID="dfb8c3c9-4f23-4f81-806a-4d89eab16c48" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:29:36.615006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:36.614912    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:29:36.615152 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:36.615001    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7" podUID="e418db77-2841-458b-be0e-e9c69ee81ac1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:29:37.813517 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:37.813488    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"
Nov 02 12:29:37.813941 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:37.813560    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"
Nov 02 12:29:38.989603 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:38.989571    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"]
Nov 02 12:29:38.993572 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:38.993554    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"
Nov 02 12:29:39.044952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.044920    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30bd0275-47hsq\" (UID: \"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"
Nov 02 12:29:39.045135 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.044960    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm8fv\" (UniqueName: \"kubernetes.io/projected/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-kube-api-access-dm8fv\") pod \"maas-api-revoke-sub-30bd0275-47hsq\" (UID: \"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"
Nov 02 12:29:39.045180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.045155    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30bd0275-47hsq\" (UID: \"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"
Nov 02 12:29:39.046950 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.046924    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:29:39.050989 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.050956    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"]
Nov 02 12:29:39.097572 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.097543    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:29:39.097731 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.097572    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:29:39.097731 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.097628    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:29:39.146345 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.146315    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30bd0275-47hsq\" (UID: \"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"
Nov 02 12:29:39.146518 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.146355    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm8fv\" (UniqueName: \"kubernetes.io/projected/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-kube-api-access-dm8fv\") pod \"maas-api-revoke-sub-30bd0275-47hsq\" (UID: \"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"
Nov 02 12:29:39.146518 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.146506    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30bd0275-47hsq\" (UID: \"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"
Nov 02 12:29:39.147110 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.147088    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30bd0275-47hsq\" (UID: \"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"
Nov 02 12:29:39.148765 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.148746    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30bd0275-47hsq\" (UID: \"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"
Nov 02 12:29:39.208586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.208563    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm8fv\" (UniqueName: \"kubernetes.io/projected/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-kube-api-access-dm8fv\") pod \"maas-api-revoke-sub-30bd0275-47hsq\" (UID: \"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"
Nov 02 12:29:39.307458 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.307401    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"
Nov 02 12:29:39.684000 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:29:39.683961    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ae1bdd_cf69_4dcc_a2a6_6495667e0bea.slice/crio-7677582c7eda37897cc8c997bc38e5d55f001f8e7eb7d3d9b46d9041c316acf0 WatchSource:0}: Error finding container 7677582c7eda37897cc8c997bc38e5d55f001f8e7eb7d3d9b46d9041c316acf0: Status 404 returned error can't find the container with id 7677582c7eda37897cc8c997bc38e5d55f001f8e7eb7d3d9b46d9041c316acf0
Nov 02 12:29:39.684427 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.684399    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"]
Nov 02 12:29:39.938601 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.938568    4464 generic.go:356] "Generic (PLEG): container finished" podID="dfb8c3c9-4f23-4f81-806a-4d89eab16c48" containerID="ce6c1613298834fd7d77d28dbe0031064629b693c74cfbbe2956fc4dd171acbd" exitCode=0
Nov 02 12:29:39.938725 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.938664    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7" event={"ID":"dfb8c3c9-4f23-4f81-806a-4d89eab16c48","Type":"ContainerDied","Data":"ce6c1613298834fd7d77d28dbe0031064629b693c74cfbbe2956fc4dd171acbd"}
Nov 02 12:29:39.940802 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.940780    4464 generic.go:356] "Generic (PLEG): container finished" podID="e418db77-2841-458b-be0e-e9c69ee81ac1" containerID="f9000f5976de400442688fe5b64c5b72fb62093139dacc38abeba370b129c778" exitCode=0
Nov 02 12:29:39.940916 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.940849    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7" event={"ID":"e418db77-2841-458b-be0e-e9c69ee81ac1","Type":"ContainerDied","Data":"f9000f5976de400442688fe5b64c5b72fb62093139dacc38abeba370b129c778"}
Nov 02 12:29:39.942557 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:39.942530    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq" event={"ID":"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea","Type":"ContainerStarted","Data":"7677582c7eda37897cc8c997bc38e5d55f001f8e7eb7d3d9b46d9041c316acf0"}
Nov 02 12:29:40.076761 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.076734    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:29:40.113021 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.112998    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:29:40.158079 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.157954    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istiod-ca-cert\") pod \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") "
Nov 02 12:29:40.158079 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158023    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e418db77-2841-458b-be0e-e9c69ee81ac1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e418db77-2841-458b-be0e-e9c69ee81ac1-istiod-ca-cert\") pod \"e418db77-2841-458b-be0e-e9c69ee81ac1\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") "
Nov 02 12:29:40.158247 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158077    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-credential-socket\") pod \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") "
Nov 02 12:29:40.158247 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158112    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-envoy\") pod \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") "
Nov 02 12:29:40.158247 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158168    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-workload-certs\") pod \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") "
Nov 02 12:29:40.158247 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158203    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-token\" (UniqueName: \"kubernetes.io/projected/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-token\") pod \"e418db77-2841-458b-be0e-e9c69ee81ac1\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") "
Nov 02 12:29:40.158247 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158228    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-data\") pod \"e418db77-2841-458b-be0e-e9c69ee81ac1\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") "
Nov 02 12:29:40.158662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158267    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-podinfo\") pod \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") "
Nov 02 12:29:40.158662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158309    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-workload-socket\") pod \"e418db77-2841-458b-be0e-e9c69ee81ac1\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") "
Nov 02 12:29:40.158662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158344    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-data\") pod \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") "
Nov 02 12:29:40.158662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158385    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-credential-socket" pod "dfb8c3c9-4f23-4f81-806a-4d89eab16c48" (UID: "dfb8c3c9-4f23-4f81-806a-4d89eab16c48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:40.158662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158391    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-workload-certs\") pod \"e418db77-2841-458b-be0e-e9c69ee81ac1\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") "
Nov 02 12:29:40.158662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158424    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e418db77-2841-458b-be0e-e9c69ee81ac1-istiod-ca-cert" pod "e418db77-2841-458b-be0e-e9c69ee81ac1" (UID: "e418db77-2841-458b-be0e-e9c69ee81ac1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:29:40.158662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158524    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-credential-socket\") pod \"e418db77-2841-458b-be0e-e9c69ee81ac1\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") "
Nov 02 12:29:40.158662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158570    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e418db77-2841-458b-be0e-e9c69ee81ac1-kube-api-access-846gr\" (UniqueName: \"kubernetes.io/projected/e418db77-2841-458b-be0e-e9c69ee81ac1-kube-api-access-846gr\") pod \"e418db77-2841-458b-be0e-e9c69ee81ac1\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") "
Nov 02 12:29:40.158662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158585    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-workload-certs" pod "dfb8c3c9-4f23-4f81-806a-4d89eab16c48" (UID: "dfb8c3c9-4f23-4f81-806a-4d89eab16c48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:40.158662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158594    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istiod-ca-cert" pod "dfb8c3c9-4f23-4f81-806a-4d89eab16c48" (UID: "dfb8c3c9-4f23-4f81-806a-4d89eab16c48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:29:40.158662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158652    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-token\" (UniqueName: \"kubernetes.io/projected/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-token\") pod \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") "
Nov 02 12:29:40.159313 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158698    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-envoy\") pod \"e418db77-2841-458b-be0e-e9c69ee81ac1\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") "
Nov 02 12:29:40.159313 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158771    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-workload-socket\") pod \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") "
Nov 02 12:29:40.159313 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158864    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-kube-api-access-d4xq5\" (UniqueName: \"kubernetes.io/projected/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-kube-api-access-d4xq5\") pod \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\" (UID: \"dfb8c3c9-4f23-4f81-806a-4d89eab16c48\") "
Nov 02 12:29:40.159313 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158899    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-podinfo\") pod \"e418db77-2841-458b-be0e-e9c69ee81ac1\" (UID: \"e418db77-2841-458b-be0e-e9c69ee81ac1\") "
Nov 02 12:29:40.159313 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.158938    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-data" pod "dfb8c3c9-4f23-4f81-806a-4d89eab16c48" (UID: "dfb8c3c9-4f23-4f81-806a-4d89eab16c48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:40.159313 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.159166    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-data" pod "e418db77-2841-458b-be0e-e9c69ee81ac1" (UID: "e418db77-2841-458b-be0e-e9c69ee81ac1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:40.159730 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.159355    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-credential-socket" pod "e418db77-2841-458b-be0e-e9c69ee81ac1" (UID: "e418db77-2841-458b-be0e-e9c69ee81ac1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:40.159730 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.159506    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.159730 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.159525    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.159730 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.159540    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.159730 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.159554    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.159730 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.159569    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.159730 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.159582    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e418db77-2841-458b-be0e-e9c69ee81ac1-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.159730 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.159595    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.160638 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.160304    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-workload-socket" pod "dfb8c3c9-4f23-4f81-806a-4d89eab16c48" (UID: "dfb8c3c9-4f23-4f81-806a-4d89eab16c48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:40.160638 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.160600    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-workload-socket" pod "e418db77-2841-458b-be0e-e9c69ee81ac1" (UID: "e418db77-2841-458b-be0e-e9c69ee81ac1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:40.160817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.160793    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-workload-certs" pod "e418db77-2841-458b-be0e-e9c69ee81ac1" (UID: "e418db77-2841-458b-be0e-e9c69ee81ac1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:40.160910 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.160885    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-envoy" pod "dfb8c3c9-4f23-4f81-806a-4d89eab16c48" (UID: "dfb8c3c9-4f23-4f81-806a-4d89eab16c48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:40.161449 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.161417    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-token" pod "e418db77-2841-458b-be0e-e9c69ee81ac1" (UID: "e418db77-2841-458b-be0e-e9c69ee81ac1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:29:40.161892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.161855    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e418db77-2841-458b-be0e-e9c69ee81ac1-kube-api-access-846gr" pod "e418db77-2841-458b-be0e-e9c69ee81ac1" (UID: "e418db77-2841-458b-be0e-e9c69ee81ac1"). InnerVolumeSpecName "kube-api-access-846gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:29:40.162052 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.161896    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-token" pod "dfb8c3c9-4f23-4f81-806a-4d89eab16c48" (UID: "dfb8c3c9-4f23-4f81-806a-4d89eab16c48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:29:40.162391 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.162369    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-kube-api-access-d4xq5" pod "dfb8c3c9-4f23-4f81-806a-4d89eab16c48" (UID: "dfb8c3c9-4f23-4f81-806a-4d89eab16c48"). InnerVolumeSpecName "kube-api-access-d4xq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:29:40.162934 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.162908    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-podinfo" pod "e418db77-2841-458b-be0e-e9c69ee81ac1" (UID: "e418db77-2841-458b-be0e-e9c69ee81ac1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:29:40.164084 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.164061    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-envoy" pod "e418db77-2841-458b-be0e-e9c69ee81ac1" (UID: "e418db77-2841-458b-be0e-e9c69ee81ac1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:40.164168 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.164108    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-podinfo" pod "dfb8c3c9-4f23-4f81-806a-4d89eab16c48" (UID: "dfb8c3c9-4f23-4f81-806a-4d89eab16c48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:29:40.260670 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.260641    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.260670 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.260671    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-846gr\" (UniqueName: \"kubernetes.io/projected/e418db77-2841-458b-be0e-e9c69ee81ac1-kube-api-access-846gr\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.260884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.260685    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.260884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.260697    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.260884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.260711    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.260884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.260724    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4xq5\" (UniqueName: \"kubernetes.io/projected/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-kube-api-access-d4xq5\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.260884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.260737    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.260884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.260751    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.260884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.260763    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e418db77-2841-458b-be0e-e9c69ee81ac1-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.260884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.260776    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb8c3c9-4f23-4f81-806a-4d89eab16c48-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.260884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.260788    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e418db77-2841-458b-be0e-e9c69ee81ac1-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:40.950583 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.950496    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"
Nov 02 12:29:40.950583 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.950505    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7" event={"ID":"e418db77-2841-458b-be0e-e9c69ee81ac1","Type":"ContainerDied","Data":"0e9c91b895d66a3ec62dcf0ad65c787bc1582cff8aa43045c8ecb7c0ee0376fd"}
Nov 02 12:29:40.950803 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.950593    4464 scope.go:122] "RemoveContainer" containerID="f9000f5976de400442688fe5b64c5b72fb62093139dacc38abeba370b129c778"
Nov 02 12:29:40.952250 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.952223    4464 generic.go:356] "Generic (PLEG): container finished" podID="b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea" containerID="82c77c83ea8adc7c600be9f2048d13d8cb3f66052747417c25669ff4a705561c" exitCode=0
Nov 02 12:29:40.952370 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.952302    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq" event={"ID":"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea","Type":"ContainerDied","Data":"82c77c83ea8adc7c600be9f2048d13d8cb3f66052747417c25669ff4a705561c"}
Nov 02 12:29:40.953786 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.953765    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7" event={"ID":"dfb8c3c9-4f23-4f81-806a-4d89eab16c48","Type":"ContainerDied","Data":"4a497081055e2fb520dae8a2bda6ab4aa2c246dc11ee07723e30839197c770ae"}
Nov 02 12:29:40.953887 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.953821    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"
Nov 02 12:29:40.961845 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:40.961827    4464 scope.go:122] "RemoveContainer" containerID="ce6c1613298834fd7d77d28dbe0031064629b693c74cfbbe2956fc4dd171acbd"
Nov 02 12:29:41.171640 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:41.171603    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"]
Nov 02 12:29:41.222538 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:41.222495    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-gc9v7"]
Nov 02 12:29:41.334224 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:41.334189    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"]
Nov 02 12:29:41.338228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:41.338202    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-h4xv7"]
Nov 02 12:29:41.582551 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:41.582474    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfb8c3c9-4f23-4f81-806a-4d89eab16c48" path="/var/lib/kubelet/pods/dfb8c3c9-4f23-4f81-806a-4d89eab16c48/volumes"
Nov 02 12:29:41.582929 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:41.582915    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e418db77-2841-458b-be0e-e9c69ee81ac1" path="/var/lib/kubelet/pods/e418db77-2841-458b-be0e-e9c69ee81ac1/volumes"
Nov 02 12:29:42.117333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:42.117309    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"
Nov 02 12:29:42.187154 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:42.187131    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-kube-api-access-dm8fv\" (UniqueName: \"kubernetes.io/projected/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-kube-api-access-dm8fv\") pod \"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea\" (UID: \"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea\") "
Nov 02 12:29:42.187606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:42.187168    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-maas-api-xks-ca\") pod \"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea\" (UID: \"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea\") "
Nov 02 12:29:42.187606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:42.187302    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-maas-api-openshift-ca\") pod \"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea\" (UID: \"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea\") "
Nov 02 12:29:42.187704 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:42.187669    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-maas-api-openshift-ca" pod "b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea" (UID: "b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:29:42.187750 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:42.187735    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:42.189232 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:42.189204    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-maas-api-xks-ca" pod "b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea" (UID: "b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:29:42.189347 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:42.189309    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-kube-api-access-dm8fv" pod "b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea" (UID: "b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea"). InnerVolumeSpecName "kube-api-access-dm8fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:29:42.288686 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:42.288654    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm8fv\" (UniqueName: \"kubernetes.io/projected/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-kube-api-access-dm8fv\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:42.288686 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:42.288682    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:42.972783 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:42.972750    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq" event={"ID":"b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea","Type":"ContainerDied","Data":"7677582c7eda37897cc8c997bc38e5d55f001f8e7eb7d3d9b46d9041c316acf0"}
Nov 02 12:29:42.972783 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:42.972778    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"
Nov 02 12:29:42.973038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:42.972808    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7677582c7eda37897cc8c997bc38e5d55f001f8e7eb7d3d9b46d9041c316acf0"
Nov 02 12:29:47.745000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:47.744047    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-62g8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:29:47.745000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:47.744106    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c" podUID="896e3cac-30cf-492b-a20a-cec437dc6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:29:47.777394 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:47.769096    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jthbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:29:47.777394 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:47.769160    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv" podUID="890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:29:49.736903 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.736864    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"]
Nov 02 12:29:49.737680 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.737657    4464 cpu_manager.go:414] "containerMap: removing container" podUID="b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea" containerName="revoke-keys"
Nov 02 12:29:49.737791 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.737682    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea" containerName="revoke-keys"
Nov 02 12:29:49.737791 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.737732    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e418db77-2841-458b-be0e-e9c69ee81ac1" containerName="istio-proxy"
Nov 02 12:29:49.737791 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.737741    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e418db77-2841-458b-be0e-e9c69ee81ac1" containerName="istio-proxy"
Nov 02 12:29:49.737791 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.737766    4464 cpu_manager.go:414] "containerMap: removing container" podUID="dfb8c3c9-4f23-4f81-806a-4d89eab16c48" containerName="istio-proxy"
Nov 02 12:29:49.737791 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.737773    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb8c3c9-4f23-4f81-806a-4d89eab16c48" containerName="istio-proxy"
Nov 02 12:29:49.738080 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.737914    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e418db77-2841-458b-be0e-e9c69ee81ac1" containerName="istio-proxy"
Nov 02 12:29:49.738080 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.737929    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb8c3c9-4f23-4f81-806a-4d89eab16c48" containerName="istio-proxy"
Nov 02 12:29:49.738080 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.737945    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea" containerName="revoke-keys"
Nov 02 12:29:49.741750 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.741729    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"
Nov 02 12:29:49.795926 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.795851    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:29:49.842388 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.842355    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt26t\" (UniqueName: \"kubernetes.io/projected/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-kube-api-access-jt26t\") pod \"maas-api-revoke-sub-f1cd40b6-flc24\" (UID: \"f09f75fe-08ae-4c85-9bbc-62aae816d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"
Nov 02 12:29:49.842549 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.842399    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1cd40b6-flc24\" (UID: \"f09f75fe-08ae-4c85-9bbc-62aae816d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"
Nov 02 12:29:49.842630 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.842594    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1cd40b6-flc24\" (UID: \"f09f75fe-08ae-4c85-9bbc-62aae816d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"
Nov 02 12:29:49.846389 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.846362    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:29:49.846515 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.846452    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:29:49.846612 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.846596    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:29:49.852114 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.852089    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"]
Nov 02 12:29:49.944102 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.944061    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt26t\" (UniqueName: \"kubernetes.io/projected/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-kube-api-access-jt26t\") pod \"maas-api-revoke-sub-f1cd40b6-flc24\" (UID: \"f09f75fe-08ae-4c85-9bbc-62aae816d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"
Nov 02 12:29:49.944285 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.944116    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1cd40b6-flc24\" (UID: \"f09f75fe-08ae-4c85-9bbc-62aae816d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"
Nov 02 12:29:49.944285 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.944261    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1cd40b6-flc24\" (UID: \"f09f75fe-08ae-4c85-9bbc-62aae816d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"
Nov 02 12:29:49.944805 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.944781    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1cd40b6-flc24\" (UID: \"f09f75fe-08ae-4c85-9bbc-62aae816d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"
Nov 02 12:29:49.946735 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:49.946709    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1cd40b6-flc24\" (UID: \"f09f75fe-08ae-4c85-9bbc-62aae816d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"
Nov 02 12:29:50.006259 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:50.006190    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt26t\" (UniqueName: \"kubernetes.io/projected/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-kube-api-access-jt26t\") pod \"maas-api-revoke-sub-f1cd40b6-flc24\" (UID: \"f09f75fe-08ae-4c85-9bbc-62aae816d7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"
Nov 02 12:29:50.055832 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:50.055800    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"
Nov 02 12:29:50.431871 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:50.431846    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"]
Nov 02 12:29:50.432207 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:29:50.432179    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09f75fe_08ae_4c85_9bbc_62aae816d7c3.slice/crio-a6360ef2978768ac29ba7ca1f42252a5436daedd06ebb84be9bc02ae1734995a WatchSource:0}: Error finding container a6360ef2978768ac29ba7ca1f42252a5436daedd06ebb84be9bc02ae1734995a: Status 404 returned error can't find the container with id a6360ef2978768ac29ba7ca1f42252a5436daedd06ebb84be9bc02ae1734995a
Nov 02 12:29:51.100433 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:51.100343    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24" event={"ID":"f09f75fe-08ae-4c85-9bbc-62aae816d7c3","Type":"ContainerStarted","Data":"4f026e2914df704f2fbcfeaeb319eb14b291fda80e7117e737f49ed9382c08f0"}
Nov 02 12:29:51.100433 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:51.100418    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24" event={"ID":"f09f75fe-08ae-4c85-9bbc-62aae816d7c3","Type":"ContainerStarted","Data":"a6360ef2978768ac29ba7ca1f42252a5436daedd06ebb84be9bc02ae1734995a"}
Nov 02 12:29:51.270298 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:51.270246    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24" podStartSLOduration=2.270232188 podStartE2EDuration="2.270232188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:29:51.218269198 +0000 UTC m=+2019.781977619" watchObservedRunningTime="2026-11-02 12:29:51.270232188 +0000 UTC m=+2019.833940596"
Nov 02 12:29:52.109039 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:52.109005    4464 generic.go:356] "Generic (PLEG): container finished" podID="f09f75fe-08ae-4c85-9bbc-62aae816d7c3" containerID="4f026e2914df704f2fbcfeaeb319eb14b291fda80e7117e737f49ed9382c08f0" exitCode=0
Nov 02 12:29:52.109412 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:52.109080    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24" event={"ID":"f09f75fe-08ae-4c85-9bbc-62aae816d7c3","Type":"ContainerDied","Data":"4f026e2914df704f2fbcfeaeb319eb14b291fda80e7117e737f49ed9382c08f0"}
Nov 02 12:29:53.273385 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:53.273361    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"
Nov 02 12:29:53.388999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:53.388894    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-maas-api-xks-ca\") pod \"f09f75fe-08ae-4c85-9bbc-62aae816d7c3\" (UID: \"f09f75fe-08ae-4c85-9bbc-62aae816d7c3\") "
Nov 02 12:29:53.389160 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:53.389060    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-kube-api-access-jt26t\" (UniqueName: \"kubernetes.io/projected/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-kube-api-access-jt26t\") pod \"f09f75fe-08ae-4c85-9bbc-62aae816d7c3\" (UID: \"f09f75fe-08ae-4c85-9bbc-62aae816d7c3\") "
Nov 02 12:29:53.389160 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:53.389087    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-maas-api-openshift-ca\") pod \"f09f75fe-08ae-4c85-9bbc-62aae816d7c3\" (UID: \"f09f75fe-08ae-4c85-9bbc-62aae816d7c3\") "
Nov 02 12:29:53.389514 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:53.389482    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-maas-api-openshift-ca" pod "f09f75fe-08ae-4c85-9bbc-62aae816d7c3" (UID: "f09f75fe-08ae-4c85-9bbc-62aae816d7c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:29:53.391664 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:53.391636    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-maas-api-xks-ca" pod "f09f75fe-08ae-4c85-9bbc-62aae816d7c3" (UID: "f09f75fe-08ae-4c85-9bbc-62aae816d7c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:29:53.391820 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:53.391779    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-kube-api-access-jt26t" pod "f09f75fe-08ae-4c85-9bbc-62aae816d7c3" (UID: "f09f75fe-08ae-4c85-9bbc-62aae816d7c3"). InnerVolumeSpecName "kube-api-access-jt26t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:29:53.490508 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:53.490474    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:53.490508 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:53.490506    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt26t\" (UniqueName: \"kubernetes.io/projected/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-kube-api-access-jt26t\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:53.490725 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:53.490520    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f09f75fe-08ae-4c85-9bbc-62aae816d7c3-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:54.125643 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:54.125613    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"
Nov 02 12:29:54.125846 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:54.125654    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24" event={"ID":"f09f75fe-08ae-4c85-9bbc-62aae816d7c3","Type":"ContainerDied","Data":"a6360ef2978768ac29ba7ca1f42252a5436daedd06ebb84be9bc02ae1734995a"}
Nov 02 12:29:54.125846 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:54.125708    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6360ef2978768ac29ba7ca1f42252a5436daedd06ebb84be9bc02ae1734995a"
Nov 02 12:29:56.053792 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.053700    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/models-as-a-service_payload-processing-external-model-5cc9844f6-pv7vm_1068cf8a-7fc4-48ea-9522-3fc6e914891a/payload-processing/0.log"
Nov 02 12:29:56.439508 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.439424    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"]
Nov 02 12:29:56.440057 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.440042    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f09f75fe-08ae-4c85-9bbc-62aae816d7c3" containerName="revoke-keys"
Nov 02 12:29:56.440101 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.440058    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09f75fe-08ae-4c85-9bbc-62aae816d7c3" containerName="revoke-keys"
Nov 02 12:29:56.440153 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.440144    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09f75fe-08ae-4c85-9bbc-62aae816d7c3" containerName="revoke-keys"
Nov 02 12:29:56.443753 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.443734    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"
Nov 02 12:29:56.497013 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.496987    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:29:56.497162 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.497132    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:29:56.521315 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.521287    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-554vf\" (UniqueName: \"kubernetes.io/projected/e017e5c3-9876-47ff-8c33-2d53501e03dd-kube-api-access-554vf\") pod \"maas-api-revoke-sub-7ca95735-ctrmf\" (UID: \"e017e5c3-9876-47ff-8c33-2d53501e03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"
Nov 02 12:29:56.521441 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.521340    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e017e5c3-9876-47ff-8c33-2d53501e03dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ca95735-ctrmf\" (UID: \"e017e5c3-9876-47ff-8c33-2d53501e03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"
Nov 02 12:29:56.521541 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.521522    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e017e5c3-9876-47ff-8c33-2d53501e03dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ca95735-ctrmf\" (UID: \"e017e5c3-9876-47ff-8c33-2d53501e03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"
Nov 02 12:29:56.547804 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.547783    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:29:56.548006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.547956    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:29:56.551444 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.551422    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"]
Nov 02 12:29:56.623146 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.623104    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-554vf\" (UniqueName: \"kubernetes.io/projected/e017e5c3-9876-47ff-8c33-2d53501e03dd-kube-api-access-554vf\") pod \"maas-api-revoke-sub-7ca95735-ctrmf\" (UID: \"e017e5c3-9876-47ff-8c33-2d53501e03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"
Nov 02 12:29:56.623337 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.623164    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e017e5c3-9876-47ff-8c33-2d53501e03dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ca95735-ctrmf\" (UID: \"e017e5c3-9876-47ff-8c33-2d53501e03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"
Nov 02 12:29:56.623337 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.623253    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e017e5c3-9876-47ff-8c33-2d53501e03dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ca95735-ctrmf\" (UID: \"e017e5c3-9876-47ff-8c33-2d53501e03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"
Nov 02 12:29:56.623892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.623866    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e017e5c3-9876-47ff-8c33-2d53501e03dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ca95735-ctrmf\" (UID: \"e017e5c3-9876-47ff-8c33-2d53501e03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"
Nov 02 12:29:56.625472 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.625456    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e017e5c3-9876-47ff-8c33-2d53501e03dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ca95735-ctrmf\" (UID: \"e017e5c3-9876-47ff-8c33-2d53501e03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"
Nov 02 12:29:56.718090 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.718009    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-554vf\" (UniqueName: \"kubernetes.io/projected/e017e5c3-9876-47ff-8c33-2d53501e03dd-kube-api-access-554vf\") pod \"maas-api-revoke-sub-7ca95735-ctrmf\" (UID: \"e017e5c3-9876-47ff-8c33-2d53501e03dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"
Nov 02 12:29:56.757910 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:56.757875    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"
Nov 02 12:29:57.139263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:57.139207    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"]
Nov 02 12:29:57.141961 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:29:57.141932    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode017e5c3_9876_47ff_8c33_2d53501e03dd.slice/crio-4fb0e052280f220536146394bec8b69a5df87f1443b0bc38dfb91d45230a8608 WatchSource:0}: Error finding container 4fb0e052280f220536146394bec8b69a5df87f1443b0bc38dfb91d45230a8608: Status 404 returned error can't find the container with id 4fb0e052280f220536146394bec8b69a5df87f1443b0bc38dfb91d45230a8608
Nov 02 12:29:58.155992 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:58.155950    4464 generic.go:356] "Generic (PLEG): container finished" podID="e017e5c3-9876-47ff-8c33-2d53501e03dd" containerID="0cfa22d252a6c0e3d0cb02c46e5235e27ba60440d44a2f66707fa13faec016b7" exitCode=0
Nov 02 12:29:58.156308 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:58.156035    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf" event={"ID":"e017e5c3-9876-47ff-8c33-2d53501e03dd","Type":"ContainerDied","Data":"0cfa22d252a6c0e3d0cb02c46e5235e27ba60440d44a2f66707fa13faec016b7"}
Nov 02 12:29:58.156308 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:58.156087    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf" event={"ID":"e017e5c3-9876-47ff-8c33-2d53501e03dd","Type":"ContainerStarted","Data":"4fb0e052280f220536146394bec8b69a5df87f1443b0bc38dfb91d45230a8608"}
Nov 02 12:29:58.292945 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:29:58.292852    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:29:59.319578 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:59.319553    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"
Nov 02 12:29:59.475377 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:59.475329    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e017e5c3-9876-47ff-8c33-2d53501e03dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e017e5c3-9876-47ff-8c33-2d53501e03dd-maas-api-openshift-ca\") pod \"e017e5c3-9876-47ff-8c33-2d53501e03dd\" (UID: \"e017e5c3-9876-47ff-8c33-2d53501e03dd\") "
Nov 02 12:29:59.475566 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:59.475472    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e017e5c3-9876-47ff-8c33-2d53501e03dd-kube-api-access-554vf\" (UniqueName: \"kubernetes.io/projected/e017e5c3-9876-47ff-8c33-2d53501e03dd-kube-api-access-554vf\") pod \"e017e5c3-9876-47ff-8c33-2d53501e03dd\" (UID: \"e017e5c3-9876-47ff-8c33-2d53501e03dd\") "
Nov 02 12:29:59.475566 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:59.475536    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e017e5c3-9876-47ff-8c33-2d53501e03dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e017e5c3-9876-47ff-8c33-2d53501e03dd-maas-api-xks-ca\") pod \"e017e5c3-9876-47ff-8c33-2d53501e03dd\" (UID: \"e017e5c3-9876-47ff-8c33-2d53501e03dd\") "
Nov 02 12:29:59.475726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:59.475677    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e017e5c3-9876-47ff-8c33-2d53501e03dd-maas-api-openshift-ca" pod "e017e5c3-9876-47ff-8c33-2d53501e03dd" (UID: "e017e5c3-9876-47ff-8c33-2d53501e03dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:29:59.476182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:59.476156    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e017e5c3-9876-47ff-8c33-2d53501e03dd-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:59.477614 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:59.477588    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e017e5c3-9876-47ff-8c33-2d53501e03dd-maas-api-xks-ca" pod "e017e5c3-9876-47ff-8c33-2d53501e03dd" (UID: "e017e5c3-9876-47ff-8c33-2d53501e03dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:29:59.477728 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:59.477715    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e017e5c3-9876-47ff-8c33-2d53501e03dd-kube-api-access-554vf" pod "e017e5c3-9876-47ff-8c33-2d53501e03dd" (UID: "e017e5c3-9876-47ff-8c33-2d53501e03dd"). InnerVolumeSpecName "kube-api-access-554vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:29:59.579261 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:59.578383    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e017e5c3-9876-47ff-8c33-2d53501e03dd-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:29:59.579261 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:29:59.578420    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-554vf\" (UniqueName: \"kubernetes.io/projected/e017e5c3-9876-47ff-8c33-2d53501e03dd-kube-api-access-554vf\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:00.162108 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.162068    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x"]
Nov 02 12:30:00.162886 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.162861    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e017e5c3-9876-47ff-8c33-2d53501e03dd" containerName="revoke-keys"
Nov 02 12:30:00.162886 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.162884    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e017e5c3-9876-47ff-8c33-2d53501e03dd" containerName="revoke-keys"
Nov 02 12:30:00.163101 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.163045    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e017e5c3-9876-47ff-8c33-2d53501e03dd" containerName="revoke-keys"
Nov 02 12:30:00.166827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.166804    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j"]
Nov 02 12:30:00.166968 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.166948    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x"
Nov 02 12:30:00.171556 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.171532    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64"]
Nov 02 12:30:00.171677 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.171660    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j"
Nov 02 12:30:00.172129 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.172112    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"
Nov 02 12:30:00.176427 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.176402    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf" event={"ID":"e017e5c3-9876-47ff-8c33-2d53501e03dd","Type":"ContainerDied","Data":"4fb0e052280f220536146394bec8b69a5df87f1443b0bc38dfb91d45230a8608"}
Nov 02 12:30:00.176539 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.176440    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fb0e052280f220536146394bec8b69a5df87f1443b0bc38dfb91d45230a8608"
Nov 02 12:30:00.176539 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.176508    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64"
Nov 02 12:30:00.212042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.212016    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5"]
Nov 02 12:30:00.215957 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.215940    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl"]
Nov 02 12:30:00.216068 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.216052    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5"
Nov 02 12:30:00.220668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.220646    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts"]
Nov 02 12:30:00.220820 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.220789    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl"
Nov 02 12:30:00.225057 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.225007    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x"]
Nov 02 12:30:00.225177 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.225083    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts"
Nov 02 12:30:00.265643 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.265613    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j"]
Nov 02 12:30:00.274211 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.274127    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64"]
Nov 02 12:30:00.277228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.277205    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5"]
Nov 02 12:30:00.283294 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.282596    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl"]
Nov 02 12:30:00.293105 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.293070    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts"]
Nov 02 12:30:00.300506 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.300478    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhg8n\" (UniqueName: \"kubernetes.io/projected/cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6-kube-api-access-dhg8n\") pod \"maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl\" (UID: \"cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl"
Nov 02 12:30:00.300648 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.300526    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8199902-070c-4231-b4ca-6d1a4c30403e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts\" (UID: \"b8199902-070c-4231-b4ca-6d1a4c30403e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts"
Nov 02 12:30:00.300648 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.300544    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63708af4-2ca1-4a72-94db-fe035c74534a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x\" (UID: \"63708af4-2ca1-4a72-94db-fe035c74534a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x"
Nov 02 12:30:00.300648 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.300619    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c24def35-99de-4046-a464-9758ccc8ab08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64\" (UID: \"c24def35-99de-4046-a464-9758ccc8ab08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64"
Nov 02 12:30:00.300845 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.300672    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9tk4\" (UniqueName: \"kubernetes.io/projected/21f424a5-2715-409d-8f60-f043ed4e10b4-kube-api-access-s9tk4\") pod \"maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j\" (UID: \"21f424a5-2715-409d-8f60-f043ed4e10b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j"
Nov 02 12:30:00.300845 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.300787    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvvjx\" (UniqueName: \"kubernetes.io/projected/0d2a2936-897c-4f2c-91cd-94b2142ce27b-kube-api-access-pvvjx\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5\" (UID: \"0d2a2936-897c-4f2c-91cd-94b2142ce27b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5"
Nov 02 12:30:00.300845 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.300833    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvt29\" (UniqueName: \"kubernetes.io/projected/c24def35-99de-4046-a464-9758ccc8ab08-kube-api-access-rvt29\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64\" (UID: \"c24def35-99de-4046-a464-9758ccc8ab08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64"
Nov 02 12:30:00.300970 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.300944    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbhbh\" (UniqueName: \"kubernetes.io/projected/63708af4-2ca1-4a72-94db-fe035c74534a-kube-api-access-tbhbh\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x\" (UID: \"63708af4-2ca1-4a72-94db-fe035c74534a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x"
Nov 02 12:30:00.301055 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.301001    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2a2936-897c-4f2c-91cd-94b2142ce27b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5\" (UID: \"0d2a2936-897c-4f2c-91cd-94b2142ce27b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5"
Nov 02 12:30:00.301107 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.301094    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd9sk\" (UniqueName: \"kubernetes.io/projected/b8199902-070c-4231-b4ca-6d1a4c30403e-kube-api-access-bd9sk\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts\" (UID: \"b8199902-070c-4231-b4ca-6d1a4c30403e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts"
Nov 02 12:30:00.301184 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.301167    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f424a5-2715-409d-8f60-f043ed4e10b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j\" (UID: \"21f424a5-2715-409d-8f60-f043ed4e10b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j"
Nov 02 12:30:00.301289 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.301273    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl\" (UID: \"cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl"
Nov 02 12:30:00.402548 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.402511    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f424a5-2715-409d-8f60-f043ed4e10b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j\" (UID: \"21f424a5-2715-409d-8f60-f043ed4e10b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j"
Nov 02 12:30:00.403081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.402604    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl\" (UID: \"cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl"
Nov 02 12:30:00.403081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.402660    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhg8n\" (UniqueName: \"kubernetes.io/projected/cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6-kube-api-access-dhg8n\") pod \"maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl\" (UID: \"cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl"
Nov 02 12:30:00.403081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.402712    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8199902-070c-4231-b4ca-6d1a4c30403e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts\" (UID: \"b8199902-070c-4231-b4ca-6d1a4c30403e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts"
Nov 02 12:30:00.403081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.402740    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63708af4-2ca1-4a72-94db-fe035c74534a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x\" (UID: \"63708af4-2ca1-4a72-94db-fe035c74534a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x"
Nov 02 12:30:00.403081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.402768    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c24def35-99de-4046-a464-9758ccc8ab08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64\" (UID: \"c24def35-99de-4046-a464-9758ccc8ab08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64"
Nov 02 12:30:00.403081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.402803    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9tk4\" (UniqueName: \"kubernetes.io/projected/21f424a5-2715-409d-8f60-f043ed4e10b4-kube-api-access-s9tk4\") pod \"maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j\" (UID: \"21f424a5-2715-409d-8f60-f043ed4e10b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j"
Nov 02 12:30:00.403081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.402875    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvvjx\" (UniqueName: \"kubernetes.io/projected/0d2a2936-897c-4f2c-91cd-94b2142ce27b-kube-api-access-pvvjx\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5\" (UID: \"0d2a2936-897c-4f2c-91cd-94b2142ce27b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5"
Nov 02 12:30:00.403081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.402909    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvt29\" (UniqueName: \"kubernetes.io/projected/c24def35-99de-4046-a464-9758ccc8ab08-kube-api-access-rvt29\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64\" (UID: \"c24def35-99de-4046-a464-9758ccc8ab08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64"
Nov 02 12:30:00.403081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.402948    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbhbh\" (UniqueName: \"kubernetes.io/projected/63708af4-2ca1-4a72-94db-fe035c74534a-kube-api-access-tbhbh\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x\" (UID: \"63708af4-2ca1-4a72-94db-fe035c74534a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x"
Nov 02 12:30:00.403081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.403002    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2a2936-897c-4f2c-91cd-94b2142ce27b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5\" (UID: \"0d2a2936-897c-4f2c-91cd-94b2142ce27b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5"
Nov 02 12:30:00.403081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.403042    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd9sk\" (UniqueName: \"kubernetes.io/projected/b8199902-070c-4231-b4ca-6d1a4c30403e-kube-api-access-bd9sk\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts\" (UID: \"b8199902-070c-4231-b4ca-6d1a4c30403e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts"
Nov 02 12:30:00.403700 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.403344    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f424a5-2715-409d-8f60-f043ed4e10b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j\" (UID: \"21f424a5-2715-409d-8f60-f043ed4e10b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j"
Nov 02 12:30:00.403700 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.403528    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63708af4-2ca1-4a72-94db-fe035c74534a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x\" (UID: \"63708af4-2ca1-4a72-94db-fe035c74534a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x"
Nov 02 12:30:00.403700 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.403552    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl\" (UID: \"cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl"
Nov 02 12:30:00.403700 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.403552    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8199902-070c-4231-b4ca-6d1a4c30403e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts\" (UID: \"b8199902-070c-4231-b4ca-6d1a4c30403e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts"
Nov 02 12:30:00.403700 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.403601    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c24def35-99de-4046-a464-9758ccc8ab08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64\" (UID: \"c24def35-99de-4046-a464-9758ccc8ab08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64"
Nov 02 12:30:00.404065 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.404045    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2a2936-897c-4f2c-91cd-94b2142ce27b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5\" (UID: \"0d2a2936-897c-4f2c-91cd-94b2142ce27b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5"
Nov 02 12:30:00.467150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.467082    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhg8n\" (UniqueName: \"kubernetes.io/projected/cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6-kube-api-access-dhg8n\") pod \"maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl\" (UID: \"cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl"
Nov 02 12:30:00.470234 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.470204    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbhbh\" (UniqueName: \"kubernetes.io/projected/63708af4-2ca1-4a72-94db-fe035c74534a-kube-api-access-tbhbh\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x\" (UID: \"63708af4-2ca1-4a72-94db-fe035c74534a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x"
Nov 02 12:30:00.472086 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.472059    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9tk4\" (UniqueName: \"kubernetes.io/projected/21f424a5-2715-409d-8f60-f043ed4e10b4-kube-api-access-s9tk4\") pod \"maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j\" (UID: \"21f424a5-2715-409d-8f60-f043ed4e10b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j"
Nov 02 12:30:00.472188 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.472086    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvvjx\" (UniqueName: \"kubernetes.io/projected/0d2a2936-897c-4f2c-91cd-94b2142ce27b-kube-api-access-pvvjx\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5\" (UID: \"0d2a2936-897c-4f2c-91cd-94b2142ce27b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5"
Nov 02 12:30:00.472188 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.472150    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvt29\" (UniqueName: \"kubernetes.io/projected/c24def35-99de-4046-a464-9758ccc8ab08-kube-api-access-rvt29\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64\" (UID: \"c24def35-99de-4046-a464-9758ccc8ab08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64"
Nov 02 12:30:00.473143 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.473121    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd9sk\" (UniqueName: \"kubernetes.io/projected/b8199902-070c-4231-b4ca-6d1a4c30403e-kube-api-access-bd9sk\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts\" (UID: \"b8199902-070c-4231-b4ca-6d1a4c30403e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts"
Nov 02 12:30:00.485827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.485808    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x"
Nov 02 12:30:00.493127 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.493106    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j"
Nov 02 12:30:00.500046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.500025    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64"
Nov 02 12:30:00.540413 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.540199    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5"
Nov 02 12:30:00.544964 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.544862    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl"
Nov 02 12:30:00.554164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.553964    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts"
Nov 02 12:30:00.874114 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:30:00.874087    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63708af4_2ca1_4a72_94db_fe035c74534a.slice/crio-de146efadbc497e6a7b9505e04e1bace573b9bb411460751bf680e1f5e9b5d00 WatchSource:0}: Error finding container de146efadbc497e6a7b9505e04e1bace573b9bb411460751bf680e1f5e9b5d00: Status 404 returned error can't find the container with id de146efadbc497e6a7b9505e04e1bace573b9bb411460751bf680e1f5e9b5d00
Nov 02 12:30:00.874278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.874258    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x"]
Nov 02 12:30:00.919939 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:30:00.919916    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24def35_99de_4046_a464_9758ccc8ab08.slice/crio-6531fd94a26f6b2cf39d92b9dac1ddae39d58e1c154abb6136f38925ef8e35d3 WatchSource:0}: Error finding container 6531fd94a26f6b2cf39d92b9dac1ddae39d58e1c154abb6136f38925ef8e35d3: Status 404 returned error can't find the container with id 6531fd94a26f6b2cf39d92b9dac1ddae39d58e1c154abb6136f38925ef8e35d3
Nov 02 12:30:00.920487 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.920459    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64"]
Nov 02 12:30:00.987984 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:00.987947    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts"]
Nov 02 12:30:00.988297 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:30:00.988274    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8199902_070c_4231_b4ca_6d1a4c30403e.slice/crio-9f43185afd58d55a33363fdcf932cb089f3ee29de4ace18363d6a3156ec6bf93 WatchSource:0}: Error finding container 9f43185afd58d55a33363fdcf932cb089f3ee29de4ace18363d6a3156ec6bf93: Status 404 returned error can't find the container with id 9f43185afd58d55a33363fdcf932cb089f3ee29de4ace18363d6a3156ec6bf93
Nov 02 12:30:01.006600 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:30:01.006574    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd70f0b2_a4fd_49cf_9ac0_6a587d4ad9b6.slice/crio-764a0c09e67852b97f819a06392b0415c56fb22945424d115e9dcbf51aaf24c6 WatchSource:0}: Error finding container 764a0c09e67852b97f819a06392b0415c56fb22945424d115e9dcbf51aaf24c6: Status 404 returned error can't find the container with id 764a0c09e67852b97f819a06392b0415c56fb22945424d115e9dcbf51aaf24c6
Nov 02 12:30:01.007738 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:01.007716    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl"]
Nov 02 12:30:01.009835 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:30:01.009802    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2a2936_897c_4f2c_91cd_94b2142ce27b.slice/crio-16779a6bad0f60612af0623f76d4f5fa4248c741b184622a878a164b997be64e WatchSource:0}: Error finding container 16779a6bad0f60612af0623f76d4f5fa4248c741b184622a878a164b997be64e: Status 404 returned error can't find the container with id 16779a6bad0f60612af0623f76d4f5fa4248c741b184622a878a164b997be64e
Nov 02 12:30:01.011504 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:01.011481    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5"]
Nov 02 12:30:01.039143 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:01.039120    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j"]
Nov 02 12:30:01.043199 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:30:01.043172    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f424a5_2715_409d_8f60_f043ed4e10b4.slice/crio-cc2024ec5f92050bc3107168be29e3a2792c1dc24794898eba0bd8db9cf9b81f WatchSource:0}: Error finding container cc2024ec5f92050bc3107168be29e3a2792c1dc24794898eba0bd8db9cf9b81f: Status 404 returned error can't find the container with id cc2024ec5f92050bc3107168be29e3a2792c1dc24794898eba0bd8db9cf9b81f
Nov 02 12:30:01.180720 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:01.180688    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts" event={"ID":"b8199902-070c-4231-b4ca-6d1a4c30403e","Type":"ContainerStarted","Data":"9f43185afd58d55a33363fdcf932cb089f3ee29de4ace18363d6a3156ec6bf93"}
Nov 02 12:30:01.182117 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:01.182087    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5" event={"ID":"0d2a2936-897c-4f2c-91cd-94b2142ce27b","Type":"ContainerStarted","Data":"16779a6bad0f60612af0623f76d4f5fa4248c741b184622a878a164b997be64e"}
Nov 02 12:30:01.184204 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:01.183717    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j" event={"ID":"21f424a5-2715-409d-8f60-f043ed4e10b4","Type":"ContainerStarted","Data":"cc2024ec5f92050bc3107168be29e3a2792c1dc24794898eba0bd8db9cf9b81f"}
Nov 02 12:30:01.185765 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:01.185741    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl" event={"ID":"cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6","Type":"ContainerStarted","Data":"764a0c09e67852b97f819a06392b0415c56fb22945424d115e9dcbf51aaf24c6"}
Nov 02 12:30:01.187211 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:01.187165    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x" event={"ID":"63708af4-2ca1-4a72-94db-fe035c74534a","Type":"ContainerStarted","Data":"de146efadbc497e6a7b9505e04e1bace573b9bb411460751bf680e1f5e9b5d00"}
Nov 02 12:30:01.188559 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:01.188539    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64" event={"ID":"c24def35-99de-4046-a464-9758ccc8ab08","Type":"ContainerStarted","Data":"6531fd94a26f6b2cf39d92b9dac1ddae39d58e1c154abb6136f38925ef8e35d3"}
Nov 02 12:30:02.197517 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.197487    4464 generic.go:356] "Generic (PLEG): container finished" podID="cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6" containerID="0aa532cdc5d14f1063fe5d039fcd0ddd649578356cb62547a3c0875ae0ca49e4" exitCode=60
Nov 02 12:30:02.197885 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.197556    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl" event={"ID":"cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6","Type":"ContainerDied","Data":"0aa532cdc5d14f1063fe5d039fcd0ddd649578356cb62547a3c0875ae0ca49e4"}
Nov 02 12:30:02.197885 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.197840    4464 scope.go:122] "RemoveContainer" containerID="0aa532cdc5d14f1063fe5d039fcd0ddd649578356cb62547a3c0875ae0ca49e4"
Nov 02 12:30:02.199495 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.199392    4464 generic.go:356] "Generic (PLEG): container finished" podID="63708af4-2ca1-4a72-94db-fe035c74534a" containerID="4b949870a44e753805ff640c39ef350c2806ad944e4f0a0947b2543bc3ba5b2d" exitCode=60
Nov 02 12:30:02.199495 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.199456    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x" event={"ID":"63708af4-2ca1-4a72-94db-fe035c74534a","Type":"ContainerDied","Data":"4b949870a44e753805ff640c39ef350c2806ad944e4f0a0947b2543bc3ba5b2d"}
Nov 02 12:30:02.199669 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.199654    4464 scope.go:122] "RemoveContainer" containerID="4b949870a44e753805ff640c39ef350c2806ad944e4f0a0947b2543bc3ba5b2d"
Nov 02 12:30:02.202117 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.202096    4464 generic.go:356] "Generic (PLEG): container finished" podID="c24def35-99de-4046-a464-9758ccc8ab08" containerID="d1c57e1f0b5649ae0b856bee5e537873945a92fe9f008443ec5d0e1457a6aa30" exitCode=60
Nov 02 12:30:02.202211 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.202174    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64" event={"ID":"c24def35-99de-4046-a464-9758ccc8ab08","Type":"ContainerDied","Data":"d1c57e1f0b5649ae0b856bee5e537873945a92fe9f008443ec5d0e1457a6aa30"}
Nov 02 12:30:02.202395 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.202379    4464 scope.go:122] "RemoveContainer" containerID="d1c57e1f0b5649ae0b856bee5e537873945a92fe9f008443ec5d0e1457a6aa30"
Nov 02 12:30:02.203764 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.203745    4464 generic.go:356] "Generic (PLEG): container finished" podID="b8199902-070c-4231-b4ca-6d1a4c30403e" containerID="88faa9a4824144e00195908682a013744abc89a4d9acbeebcbd63af9ff4af6f4" exitCode=60
Nov 02 12:30:02.203856 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.203812    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts" event={"ID":"b8199902-070c-4231-b4ca-6d1a4c30403e","Type":"ContainerDied","Data":"88faa9a4824144e00195908682a013744abc89a4d9acbeebcbd63af9ff4af6f4"}
Nov 02 12:30:02.204042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.204021    4464 scope.go:122] "RemoveContainer" containerID="88faa9a4824144e00195908682a013744abc89a4d9acbeebcbd63af9ff4af6f4"
Nov 02 12:30:02.205522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.205502    4464 generic.go:356] "Generic (PLEG): container finished" podID="0d2a2936-897c-4f2c-91cd-94b2142ce27b" containerID="29f13b1e4f2d68ef92324aae7e8e3f2df009862392f10327f82c0394fb9d855a" exitCode=60
Nov 02 12:30:02.205630 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.205608    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5" event={"ID":"0d2a2936-897c-4f2c-91cd-94b2142ce27b","Type":"ContainerDied","Data":"29f13b1e4f2d68ef92324aae7e8e3f2df009862392f10327f82c0394fb9d855a"}
Nov 02 12:30:02.205740 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.205727    4464 scope.go:122] "RemoveContainer" containerID="29f13b1e4f2d68ef92324aae7e8e3f2df009862392f10327f82c0394fb9d855a"
Nov 02 12:30:02.207144 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.207126    4464 generic.go:356] "Generic (PLEG): container finished" podID="21f424a5-2715-409d-8f60-f043ed4e10b4" containerID="76bb193f7dc8f986c9e7dd524aae963c63d41fb157c253f25644eb31758d8055" exitCode=7
Nov 02 12:30:02.207238 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.207157    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j" event={"ID":"21f424a5-2715-409d-8f60-f043ed4e10b4","Type":"ContainerDied","Data":"76bb193f7dc8f986c9e7dd524aae963c63d41fb157c253f25644eb31758d8055"}
Nov 02 12:30:02.207450 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:02.207429    4464 scope.go:122] "RemoveContainer" containerID="76bb193f7dc8f986c9e7dd524aae963c63d41fb157c253f25644eb31758d8055"
Nov 02 12:30:03.218073 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:03.218046    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5" event={"ID":"0d2a2936-897c-4f2c-91cd-94b2142ce27b","Type":"ContainerStarted","Data":"4d35117e770dae9e4f64801c6a54ba671f2000b56e40dad532a83fc52f12d17d"}
Nov 02 12:30:03.219959 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:03.219934    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j" event={"ID":"21f424a5-2715-409d-8f60-f043ed4e10b4","Type":"ContainerStarted","Data":"e41bfdaeb785fde192e7e56f880522aab9df20b467b5e8e55613752d63848eaf"}
Nov 02 12:30:03.221834 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:03.221813    4464 generic.go:356] "Generic (PLEG): container finished" podID="cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6" containerID="d7fcc1b2d9ee0e75b625843118d2c4236c70badebebe6d812d6c22ba7dc41826" exitCode=60
Nov 02 12:30:03.221937 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:03.221837    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl" event={"ID":"cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6","Type":"ContainerDied","Data":"d7fcc1b2d9ee0e75b625843118d2c4236c70badebebe6d812d6c22ba7dc41826"}
Nov 02 12:30:03.221937 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:03.221891    4464 scope.go:122] "RemoveContainer" containerID="0aa532cdc5d14f1063fe5d039fcd0ddd649578356cb62547a3c0875ae0ca49e4"
Nov 02 12:30:03.222165 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:03.222147    4464 scope.go:122] "RemoveContainer" containerID="d7fcc1b2d9ee0e75b625843118d2c4236c70badebebe6d812d6c22ba7dc41826"
Nov 02 12:30:03.222456 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:03.222425    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl_odh-ai-gateway-infra(cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl" podUID="cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6"
Nov 02 12:30:03.223727 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:03.223709    4464 generic.go:356] "Generic (PLEG): container finished" podID="63708af4-2ca1-4a72-94db-fe035c74534a" containerID="4a534dd10adfab3b5812fd22413c7e3735e6a25000083155675f9b516066a23e" exitCode=60
Nov 02 12:30:03.223829 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:03.223735    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x" event={"ID":"63708af4-2ca1-4a72-94db-fe035c74534a","Type":"ContainerDied","Data":"4a534dd10adfab3b5812fd22413c7e3735e6a25000083155675f9b516066a23e"}
Nov 02 12:30:03.223966 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:03.223952    4464 scope.go:122] "RemoveContainer" containerID="4a534dd10adfab3b5812fd22413c7e3735e6a25000083155675f9b516066a23e"
Nov 02 12:30:03.224258 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:03.224236    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x_odh-ai-gateway-infra(63708af4-2ca1-4a72-94db-fe035c74534a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x" podUID="63708af4-2ca1-4a72-94db-fe035c74534a"
Nov 02 12:30:03.225577 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:03.225555    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64" event={"ID":"c24def35-99de-4046-a464-9758ccc8ab08","Type":"ContainerStarted","Data":"198b786f7b164cc4e2e612146cdae95a7c6ab62c88dfb3d6a8161fb54c198e98"}
Nov 02 12:30:03.227335 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:03.227313    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts" event={"ID":"b8199902-070c-4231-b4ca-6d1a4c30403e","Type":"ContainerStarted","Data":"b368a5dd9b2e6018afe31c53991a7977ced8d4913a84b9478b0d31b820de5484"}
Nov 02 12:30:03.236734 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:03.236715    4464 scope.go:122] "RemoveContainer" containerID="4b949870a44e753805ff640c39ef350c2806ad944e4f0a0947b2543bc3ba5b2d"
Nov 02 12:30:03.340186 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:03.340144    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5" podStartSLOduration=3.340130041 podStartE2EDuration="3.340130041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:03.337633519 +0000 UTC m=+2031.901341927" watchObservedRunningTime="2026-11-02 12:30:03.340130041 +0000 UTC m=+2031.903838449"
Nov 02 12:30:03.690069 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:03.689963    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts" podStartSLOduration=3.68995031 podStartE2EDuration="3.68995031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:03.687194448 +0000 UTC m=+2032.250902871" watchObservedRunningTime="2026-11-02 12:30:03.68995031 +0000 UTC m=+2032.253658756"
Nov 02 12:30:03.806823 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:03.806774    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j" podStartSLOduration=3.806760869 podStartE2EDuration="3.806760869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:03.804113418 +0000 UTC m=+2032.367821828" watchObservedRunningTime="2026-11-02 12:30:03.806760869 +0000 UTC m=+2032.370469302"
Nov 02 12:30:03.923205 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:03.923158    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64" podStartSLOduration=3.9231440600000003 podStartE2EDuration="3.92314406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:03.920386834 +0000 UTC m=+2032.484095247" watchObservedRunningTime="2026-11-02 12:30:03.92314406 +0000 UTC m=+2032.486852469"
Nov 02 12:30:04.241116 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.241084    4464 scope.go:122] "RemoveContainer" containerID="d7fcc1b2d9ee0e75b625843118d2c4236c70badebebe6d812d6c22ba7dc41826"
Nov 02 12:30:04.241532 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:04.241397    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl_odh-ai-gateway-infra(cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl" podUID="cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6"
Nov 02 12:30:04.242698 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.242675    4464 scope.go:122] "RemoveContainer" containerID="4a534dd10adfab3b5812fd22413c7e3735e6a25000083155675f9b516066a23e"
Nov 02 12:30:04.242957 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:04.242933    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x_odh-ai-gateway-infra(63708af4-2ca1-4a72-94db-fe035c74534a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x" podUID="63708af4-2ca1-4a72-94db-fe035c74534a"
Nov 02 12:30:04.244156 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.244132    4464 generic.go:356] "Generic (PLEG): container finished" podID="c24def35-99de-4046-a464-9758ccc8ab08" containerID="198b786f7b164cc4e2e612146cdae95a7c6ab62c88dfb3d6a8161fb54c198e98" exitCode=60
Nov 02 12:30:04.244242 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.244203    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64" event={"ID":"c24def35-99de-4046-a464-9758ccc8ab08","Type":"ContainerDied","Data":"198b786f7b164cc4e2e612146cdae95a7c6ab62c88dfb3d6a8161fb54c198e98"}
Nov 02 12:30:04.244288 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.244277    4464 scope.go:122] "RemoveContainer" containerID="d1c57e1f0b5649ae0b856bee5e537873945a92fe9f008443ec5d0e1457a6aa30"
Nov 02 12:30:04.244377 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.244360    4464 scope.go:122] "RemoveContainer" containerID="198b786f7b164cc4e2e612146cdae95a7c6ab62c88dfb3d6a8161fb54c198e98"
Nov 02 12:30:04.244618 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:04.244596    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64_odh-ai-gateway-infra(c24def35-99de-4046-a464-9758ccc8ab08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64" podUID="c24def35-99de-4046-a464-9758ccc8ab08"
Nov 02 12:30:04.246136 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.246114    4464 generic.go:356] "Generic (PLEG): container finished" podID="b8199902-070c-4231-b4ca-6d1a4c30403e" containerID="b368a5dd9b2e6018afe31c53991a7977ced8d4913a84b9478b0d31b820de5484" exitCode=60
Nov 02 12:30:04.246230 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.246178    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts" event={"ID":"b8199902-070c-4231-b4ca-6d1a4c30403e","Type":"ContainerDied","Data":"b368a5dd9b2e6018afe31c53991a7977ced8d4913a84b9478b0d31b820de5484"}
Nov 02 12:30:04.246377 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.246363    4464 scope.go:122] "RemoveContainer" containerID="b368a5dd9b2e6018afe31c53991a7977ced8d4913a84b9478b0d31b820de5484"
Nov 02 12:30:04.246596 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:04.246579    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts_odh-ai-gateway-infra(b8199902-070c-4231-b4ca-6d1a4c30403e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts" podUID="b8199902-070c-4231-b4ca-6d1a4c30403e"
Nov 02 12:30:04.247964 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.247947    4464 generic.go:356] "Generic (PLEG): container finished" podID="0d2a2936-897c-4f2c-91cd-94b2142ce27b" containerID="4d35117e770dae9e4f64801c6a54ba671f2000b56e40dad532a83fc52f12d17d" exitCode=60
Nov 02 12:30:04.248072 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.248018    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5" event={"ID":"0d2a2936-897c-4f2c-91cd-94b2142ce27b","Type":"ContainerDied","Data":"4d35117e770dae9e4f64801c6a54ba671f2000b56e40dad532a83fc52f12d17d"}
Nov 02 12:30:04.248279 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.248261    4464 scope.go:122] "RemoveContainer" containerID="4d35117e770dae9e4f64801c6a54ba671f2000b56e40dad532a83fc52f12d17d"
Nov 02 12:30:04.248511 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:04.248492    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5_odh-ai-gateway-infra(0d2a2936-897c-4f2c-91cd-94b2142ce27b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5" podUID="0d2a2936-897c-4f2c-91cd-94b2142ce27b"
Nov 02 12:30:04.249865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.249845    4464 generic.go:356] "Generic (PLEG): container finished" podID="21f424a5-2715-409d-8f60-f043ed4e10b4" containerID="e41bfdaeb785fde192e7e56f880522aab9df20b467b5e8e55613752d63848eaf" exitCode=7
Nov 02 12:30:04.249948 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.249876    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j" event={"ID":"21f424a5-2715-409d-8f60-f043ed4e10b4","Type":"ContainerDied","Data":"e41bfdaeb785fde192e7e56f880522aab9df20b467b5e8e55613752d63848eaf"}
Nov 02 12:30:04.250136 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.250120    4464 scope.go:122] "RemoveContainer" containerID="e41bfdaeb785fde192e7e56f880522aab9df20b467b5e8e55613752d63848eaf"
Nov 02 12:30:04.250372 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:04.250352    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j_odh-ai-gateway-infra(21f424a5-2715-409d-8f60-f043ed4e10b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j" podUID="21f424a5-2715-409d-8f60-f043ed4e10b4"
Nov 02 12:30:04.257193 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.257168    4464 scope.go:122] "RemoveContainer" containerID="88faa9a4824144e00195908682a013744abc89a4d9acbeebcbd63af9ff4af6f4"
Nov 02 12:30:04.269660 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.269641    4464 scope.go:122] "RemoveContainer" containerID="29f13b1e4f2d68ef92324aae7e8e3f2df009862392f10327f82c0394fb9d855a"
Nov 02 12:30:04.281945 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.281928    4464 scope.go:122] "RemoveContainer" containerID="76bb193f7dc8f986c9e7dd524aae963c63d41fb157c253f25644eb31758d8055"
Nov 02 12:30:04.665346 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:04.665251    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:30:05.258701 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:05.258673    4464 scope.go:122] "RemoveContainer" containerID="4d35117e770dae9e4f64801c6a54ba671f2000b56e40dad532a83fc52f12d17d"
Nov 02 12:30:05.259188 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:05.259013    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5_odh-ai-gateway-infra(0d2a2936-897c-4f2c-91cd-94b2142ce27b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5" podUID="0d2a2936-897c-4f2c-91cd-94b2142ce27b"
Nov 02 12:30:05.261040 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:05.261016    4464 scope.go:122] "RemoveContainer" containerID="e41bfdaeb785fde192e7e56f880522aab9df20b467b5e8e55613752d63848eaf"
Nov 02 12:30:05.261452 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:05.261423    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j_odh-ai-gateway-infra(21f424a5-2715-409d-8f60-f043ed4e10b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j" podUID="21f424a5-2715-409d-8f60-f043ed4e10b4"
Nov 02 12:30:05.264215 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:05.264187    4464 scope.go:122] "RemoveContainer" containerID="198b786f7b164cc4e2e612146cdae95a7c6ab62c88dfb3d6a8161fb54c198e98"
Nov 02 12:30:05.264540 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:05.264515    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64_odh-ai-gateway-infra(c24def35-99de-4046-a464-9758ccc8ab08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64" podUID="c24def35-99de-4046-a464-9758ccc8ab08"
Nov 02 12:30:05.266723 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:05.266701    4464 scope.go:122] "RemoveContainer" containerID="b368a5dd9b2e6018afe31c53991a7977ced8d4913a84b9478b0d31b820de5484"
Nov 02 12:30:05.266954 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:05.266930    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts_odh-ai-gateway-infra(b8199902-070c-4231-b4ca-6d1a4c30403e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts" podUID="b8199902-070c-4231-b4ca-6d1a4c30403e"
Nov 02 12:30:10.080170 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:10.080135    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"]
Nov 02 12:30:10.084575 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:10.084553    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"
Nov 02 12:30:10.136469 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:10.136443    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/155887c3-90e0-45dc-9ae2-22c0469dfef6-provider-credentials\") pod \"payload-processing-external-model-7bc9c479f4-df544\" (UID: \"155887c3-90e0-45dc-9ae2-22c0469dfef6\") " pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"
Nov 02 12:30:10.136579 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:10.136498    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/155887c3-90e0-45dc-9ae2-22c0469dfef6-plugins-config-volume\") pod \"payload-processing-external-model-7bc9c479f4-df544\" (UID: \"155887c3-90e0-45dc-9ae2-22c0469dfef6\") " pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"
Nov 02 12:30:10.136579 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:10.136565    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/155887c3-90e0-45dc-9ae2-22c0469dfef6-routing-overlay-volume\") pod \"payload-processing-external-model-7bc9c479f4-df544\" (UID: \"155887c3-90e0-45dc-9ae2-22c0469dfef6\") " pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"
Nov 02 12:30:10.136676 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:10.136655    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5jbx\" (UniqueName: \"kubernetes.io/projected/155887c3-90e0-45dc-9ae2-22c0469dfef6-kube-api-access-j5jbx\") pod \"payload-processing-external-model-7bc9c479f4-df544\" (UID: \"155887c3-90e0-45dc-9ae2-22c0469dfef6\") " pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"
Nov 02 12:30:10.143251 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:10.143229    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"]
Nov 02 12:30:10.237427 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:10.237384    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/155887c3-90e0-45dc-9ae2-22c0469dfef6-provider-credentials\") pod \"payload-processing-external-model-7bc9c479f4-df544\" (UID: \"155887c3-90e0-45dc-9ae2-22c0469dfef6\") " pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"
Nov 02 12:30:10.237581 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:10.237451    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/155887c3-90e0-45dc-9ae2-22c0469dfef6-plugins-config-volume\") pod \"payload-processing-external-model-7bc9c479f4-df544\" (UID: \"155887c3-90e0-45dc-9ae2-22c0469dfef6\") " pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"
Nov 02 12:30:10.237629 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:10.237605    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/155887c3-90e0-45dc-9ae2-22c0469dfef6-routing-overlay-volume\") pod \"payload-processing-external-model-7bc9c479f4-df544\" (UID: \"155887c3-90e0-45dc-9ae2-22c0469dfef6\") " pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"
Nov 02 12:30:10.237740 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:10.237718    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5jbx\" (UniqueName: \"kubernetes.io/projected/155887c3-90e0-45dc-9ae2-22c0469dfef6-kube-api-access-j5jbx\") pod \"payload-processing-external-model-7bc9c479f4-df544\" (UID: \"155887c3-90e0-45dc-9ae2-22c0469dfef6\") " pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"
Nov 02 12:30:10.238481 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:10.238444    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/155887c3-90e0-45dc-9ae2-22c0469dfef6-routing-overlay-volume\") pod \"payload-processing-external-model-7bc9c479f4-df544\" (UID: \"155887c3-90e0-45dc-9ae2-22c0469dfef6\") " pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"
Nov 02 12:30:10.238668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:10.238648    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/155887c3-90e0-45dc-9ae2-22c0469dfef6-plugins-config-volume\") pod \"payload-processing-external-model-7bc9c479f4-df544\" (UID: \"155887c3-90e0-45dc-9ae2-22c0469dfef6\") " pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"
Nov 02 12:30:10.240085 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:10.240070    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/155887c3-90e0-45dc-9ae2-22c0469dfef6-provider-credentials\") pod \"payload-processing-external-model-7bc9c479f4-df544\" (UID: \"155887c3-90e0-45dc-9ae2-22c0469dfef6\") " pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"
Nov 02 12:30:10.300318 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:10.300285    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5jbx\" (UniqueName: \"kubernetes.io/projected/155887c3-90e0-45dc-9ae2-22c0469dfef6-kube-api-access-j5jbx\") pod \"payload-processing-external-model-7bc9c479f4-df544\" (UID: \"155887c3-90e0-45dc-9ae2-22c0469dfef6\") " pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"
Nov 02 12:30:10.398233 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:10.398156    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"
Nov 02 12:30:10.803091 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:10.803059    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"]
Nov 02 12:30:10.803356 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:30:10.803332    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod155887c3_90e0_45dc_9ae2_22c0469dfef6.slice/crio-60e4d36661546f01a0fd8b684653768ff4f5943e47eeaee646413ca330de9034 WatchSource:0}: Error finding container 60e4d36661546f01a0fd8b684653768ff4f5943e47eeaee646413ca330de9034: Status 404 returned error can't find the container with id 60e4d36661546f01a0fd8b684653768ff4f5943e47eeaee646413ca330de9034
Nov 02 12:30:11.309292 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:11.309250    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544" event={"ID":"155887c3-90e0-45dc-9ae2-22c0469dfef6","Type":"ContainerStarted","Data":"4ca954cbb2a7fc6227daa3adb3da9bb256b388c8f23d485af73b9821a6265850"}
Nov 02 12:30:11.309783 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:11.309316    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544" event={"ID":"155887c3-90e0-45dc-9ae2-22c0469dfef6","Type":"ContainerStarted","Data":"60e4d36661546f01a0fd8b684653768ff4f5943e47eeaee646413ca330de9034"}
Nov 02 12:30:11.309783 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:11.309347    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"
Nov 02 12:30:11.429270 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:11.429216    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544" podStartSLOduration=1.429202656 podStartE2EDuration="1.429202656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:11.42763846 +0000 UTC m=+2039.991346868" watchObservedRunningTime="2026-11-02 12:30:11.429202656 +0000 UTC m=+2039.992911065"
Nov 02 12:30:13.314099 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:13.314068    4464 scope.go:122] "RemoveContainer" containerID="3e49e0909eb4d7496ec5d61738bae6adc52c615918f4c50df8058ad2f92f245b"
Nov 02 12:30:13.574948 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:13.574856    4464 scope.go:122] "RemoveContainer" containerID="d7fcc1b2d9ee0e75b625843118d2c4236c70badebebe6d812d6c22ba7dc41826"
Nov 02 12:30:14.332668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:14.332631    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl" event={"ID":"cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6","Type":"ContainerStarted","Data":"34092249ef306c9b76269db3382a061a11dbdbe374fc71e89e36a5cfc46fcb02"}
Nov 02 12:30:14.451901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:14.451849    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl" podStartSLOduration=14.451836585 podStartE2EDuration="14.451836585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:14.448878907 +0000 UTC m=+2043.012587317" watchObservedRunningTime="2026-11-02 12:30:14.451836585 +0000 UTC m=+2043.015544994"
Nov 02 12:30:14.575240 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:14.575213    4464 scope.go:122] "RemoveContainer" containerID="4d35117e770dae9e4f64801c6a54ba671f2000b56e40dad532a83fc52f12d17d"
Nov 02 12:30:15.342476 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:15.342439    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5" event={"ID":"0d2a2936-897c-4f2c-91cd-94b2142ce27b","Type":"ContainerStarted","Data":"401215d1249073130a7e6f9fb725540430ded5e0460a90931bbc5e3933af256d"}
Nov 02 12:30:15.344413 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:15.344388    4464 generic.go:356] "Generic (PLEG): container finished" podID="cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6" containerID="34092249ef306c9b76269db3382a061a11dbdbe374fc71e89e36a5cfc46fcb02" exitCode=60
Nov 02 12:30:15.344532 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:15.344460    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl" event={"ID":"cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6","Type":"ContainerDied","Data":"34092249ef306c9b76269db3382a061a11dbdbe374fc71e89e36a5cfc46fcb02"}
Nov 02 12:30:15.344532 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:15.344515    4464 scope.go:122] "RemoveContainer" containerID="d7fcc1b2d9ee0e75b625843118d2c4236c70badebebe6d812d6c22ba7dc41826"
Nov 02 12:30:15.344783 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:15.344760    4464 scope.go:122] "RemoveContainer" containerID="34092249ef306c9b76269db3382a061a11dbdbe374fc71e89e36a5cfc46fcb02"
Nov 02 12:30:15.345068 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:15.345047    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl_odh-ai-gateway-infra(cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl" podUID="cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6"
Nov 02 12:30:15.460670 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:15.460634    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl"]
Nov 02 12:30:15.575650 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:15.575618    4464 scope.go:122] "RemoveContainer" containerID="4a534dd10adfab3b5812fd22413c7e3735e6a25000083155675f9b516066a23e"
Nov 02 12:30:15.576293 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:15.576265    4464 scope.go:122] "RemoveContainer" containerID="198b786f7b164cc4e2e612146cdae95a7c6ab62c88dfb3d6a8161fb54c198e98"
Nov 02 12:30:16.354039 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:16.354007    4464 generic.go:356] "Generic (PLEG): container finished" podID="0d2a2936-897c-4f2c-91cd-94b2142ce27b" containerID="401215d1249073130a7e6f9fb725540430ded5e0460a90931bbc5e3933af256d" exitCode=60
Nov 02 12:30:16.354470 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:16.354050    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5" event={"ID":"0d2a2936-897c-4f2c-91cd-94b2142ce27b","Type":"ContainerDied","Data":"401215d1249073130a7e6f9fb725540430ded5e0460a90931bbc5e3933af256d"}
Nov 02 12:30:16.354470 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:16.354129    4464 scope.go:122] "RemoveContainer" containerID="4d35117e770dae9e4f64801c6a54ba671f2000b56e40dad532a83fc52f12d17d"
Nov 02 12:30:16.354633 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:16.354516    4464 scope.go:122] "RemoveContainer" containerID="401215d1249073130a7e6f9fb725540430ded5e0460a90931bbc5e3933af256d"
Nov 02 12:30:16.354896 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:16.354869    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5_odh-ai-gateway-infra(0d2a2936-897c-4f2c-91cd-94b2142ce27b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5" podUID="0d2a2936-897c-4f2c-91cd-94b2142ce27b"
Nov 02 12:30:16.358360 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:16.358326    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x" event={"ID":"63708af4-2ca1-4a72-94db-fe035c74534a","Type":"ContainerStarted","Data":"d934b827319a19ad4266a840628bc141889205ea88c3920a557a289c1614eadf"}
Nov 02 12:30:16.360577 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:16.360555    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64" event={"ID":"c24def35-99de-4046-a464-9758ccc8ab08","Type":"ContainerStarted","Data":"dece0834eba7a9523826c683f464dcf3ab615a82753083d1564c5050a8b3c2d0"}
Nov 02 12:30:16.472533 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:16.472505    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5"]
Nov 02 12:30:16.503730 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:16.503702    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl"
Nov 02 12:30:16.575894 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:16.575861    4464 scope.go:122] "RemoveContainer" containerID="b368a5dd9b2e6018afe31c53991a7977ced8d4913a84b9478b0d31b820de5484"
Nov 02 12:30:16.620192 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:16.620126    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6-kube-api-access-dhg8n\" (UniqueName: \"kubernetes.io/projected/cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6-kube-api-access-dhg8n\") pod \"cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6\" (UID: \"cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6\") "
Nov 02 12:30:16.620345 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:16.620326    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6-maas-api-service-ca\") pod \"cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6\" (UID: \"cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6\") "
Nov 02 12:30:16.620685 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:16.620659    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6-maas-api-service-ca" pod "cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6" (UID: "cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:30:16.620956 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:16.620936    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:16.622245 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:16.622225    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6-kube-api-access-dhg8n" pod "cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6" (UID: "cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6"). InnerVolumeSpecName "kube-api-access-dhg8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:16.716408 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:16.716352    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x" podStartSLOduration=16.716338885 podStartE2EDuration="16.716338885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:16.71548912 +0000 UTC m=+2045.279197541" watchObservedRunningTime="2026-11-02 12:30:16.716338885 +0000 UTC m=+2045.280047294"
Nov 02 12:30:16.722724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:16.722688    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhg8n\" (UniqueName: \"kubernetes.io/projected/cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6-kube-api-access-dhg8n\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:17.370649 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.370609    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts" event={"ID":"b8199902-070c-4231-b4ca-6d1a4c30403e","Type":"ContainerStarted","Data":"8c735366e52cdbebf8c1f48a57734500237790d79e22926fd2273e0c49113b98"}
Nov 02 12:30:17.373652 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.373632    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl"
Nov 02 12:30:17.373774 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.373662    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl" event={"ID":"cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6","Type":"ContainerDied","Data":"764a0c09e67852b97f819a06392b0415c56fb22945424d115e9dcbf51aaf24c6"}
Nov 02 12:30:17.373774 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.373711    4464 scope.go:122] "RemoveContainer" containerID="34092249ef306c9b76269db3382a061a11dbdbe374fc71e89e36a5cfc46fcb02"
Nov 02 12:30:17.375889 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.375864    4464 generic.go:356] "Generic (PLEG): container finished" podID="63708af4-2ca1-4a72-94db-fe035c74534a" containerID="d934b827319a19ad4266a840628bc141889205ea88c3920a557a289c1614eadf" exitCode=60
Nov 02 12:30:17.376000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.375930    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x" event={"ID":"63708af4-2ca1-4a72-94db-fe035c74534a","Type":"ContainerDied","Data":"d934b827319a19ad4266a840628bc141889205ea88c3920a557a289c1614eadf"}
Nov 02 12:30:17.376371 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.376354    4464 scope.go:122] "RemoveContainer" containerID="d934b827319a19ad4266a840628bc141889205ea88c3920a557a289c1614eadf"
Nov 02 12:30:17.376674 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:17.376650    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x_odh-ai-gateway-infra(63708af4-2ca1-4a72-94db-fe035c74534a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x" podUID="63708af4-2ca1-4a72-94db-fe035c74534a"
Nov 02 12:30:17.378051 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.378033    4464 generic.go:356] "Generic (PLEG): container finished" podID="c24def35-99de-4046-a464-9758ccc8ab08" containerID="dece0834eba7a9523826c683f464dcf3ab615a82753083d1564c5050a8b3c2d0" exitCode=60
Nov 02 12:30:17.378147 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.378066    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64" event={"ID":"c24def35-99de-4046-a464-9758ccc8ab08","Type":"ContainerDied","Data":"dece0834eba7a9523826c683f464dcf3ab615a82753083d1564c5050a8b3c2d0"}
Nov 02 12:30:17.378365 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.378352    4464 scope.go:122] "RemoveContainer" containerID="dece0834eba7a9523826c683f464dcf3ab615a82753083d1564c5050a8b3c2d0"
Nov 02 12:30:17.378565 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:17.378544    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64_odh-ai-gateway-infra(c24def35-99de-4046-a464-9758ccc8ab08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64" podUID="c24def35-99de-4046-a464-9758ccc8ab08"
Nov 02 12:30:17.385226 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.385202    4464 scope.go:122] "RemoveContainer" containerID="4a534dd10adfab3b5812fd22413c7e3735e6a25000083155675f9b516066a23e"
Nov 02 12:30:17.401462 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.401442    4464 scope.go:122] "RemoveContainer" containerID="198b786f7b164cc4e2e612146cdae95a7c6ab62c88dfb3d6a8161fb54c198e98"
Nov 02 12:30:17.526733 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.526709    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5"
Nov 02 12:30:17.575014 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.574969    4464 scope.go:122] "RemoveContainer" containerID="e41bfdaeb785fde192e7e56f880522aab9df20b467b5e8e55613752d63848eaf"
Nov 02 12:30:17.633827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.633770    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d2a2936-897c-4f2c-91cd-94b2142ce27b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2a2936-897c-4f2c-91cd-94b2142ce27b-maas-api-service-ca\") pod \"0d2a2936-897c-4f2c-91cd-94b2142ce27b\" (UID: \"0d2a2936-897c-4f2c-91cd-94b2142ce27b\") "
Nov 02 12:30:17.633933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.633924    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2a2936-897c-4f2c-91cd-94b2142ce27b-kube-api-access-pvvjx\" (UniqueName: \"kubernetes.io/projected/0d2a2936-897c-4f2c-91cd-94b2142ce27b-kube-api-access-pvvjx\") pod \"0d2a2936-897c-4f2c-91cd-94b2142ce27b\" (UID: \"0d2a2936-897c-4f2c-91cd-94b2142ce27b\") "
Nov 02 12:30:17.634095 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.634072    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2a2936-897c-4f2c-91cd-94b2142ce27b-maas-api-service-ca" pod "0d2a2936-897c-4f2c-91cd-94b2142ce27b" (UID: "0d2a2936-897c-4f2c-91cd-94b2142ce27b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:30:17.634405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.634384    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2a2936-897c-4f2c-91cd-94b2142ce27b-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:17.635983 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.635960    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2a2936-897c-4f2c-91cd-94b2142ce27b-kube-api-access-pvvjx" pod "0d2a2936-897c-4f2c-91cd-94b2142ce27b" (UID: "0d2a2936-897c-4f2c-91cd-94b2142ce27b"). InnerVolumeSpecName "kube-api-access-pvvjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:17.731381 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.731350    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x"]
Nov 02 12:30:17.735397 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.735376    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvvjx\" (UniqueName: \"kubernetes.io/projected/0d2a2936-897c-4f2c-91cd-94b2142ce27b-kube-api-access-pvvjx\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:17.836706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.836677    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl"]
Nov 02 12:30:17.842071 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.842040    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4b9210-29893710-rc7nl"]
Nov 02 12:30:17.845289 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:17.845262    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64"]
Nov 02 12:30:18.392285 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.392248    4464 generic.go:356] "Generic (PLEG): container finished" podID="b8199902-070c-4231-b4ca-6d1a4c30403e" containerID="8c735366e52cdbebf8c1f48a57734500237790d79e22926fd2273e0c49113b98" exitCode=60
Nov 02 12:30:18.392647 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.392322    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts" event={"ID":"b8199902-070c-4231-b4ca-6d1a4c30403e","Type":"ContainerDied","Data":"8c735366e52cdbebf8c1f48a57734500237790d79e22926fd2273e0c49113b98"}
Nov 02 12:30:18.392647 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.392393    4464 scope.go:122] "RemoveContainer" containerID="b368a5dd9b2e6018afe31c53991a7977ced8d4913a84b9478b0d31b820de5484"
Nov 02 12:30:18.392884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.392861    4464 scope.go:122] "RemoveContainer" containerID="8c735366e52cdbebf8c1f48a57734500237790d79e22926fd2273e0c49113b98"
Nov 02 12:30:18.393286 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:18.393259    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts_odh-ai-gateway-infra(b8199902-070c-4231-b4ca-6d1a4c30403e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts" podUID="b8199902-070c-4231-b4ca-6d1a4c30403e"
Nov 02 12:30:18.393952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.393935    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5"
Nov 02 12:30:18.394055 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.393956    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5" event={"ID":"0d2a2936-897c-4f2c-91cd-94b2142ce27b","Type":"ContainerDied","Data":"16779a6bad0f60612af0623f76d4f5fa4248c741b184622a878a164b997be64e"}
Nov 02 12:30:18.395998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.395951    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j" event={"ID":"21f424a5-2715-409d-8f60-f043ed4e10b4","Type":"ContainerStarted","Data":"a41b1353265f315642cc6de9328d758f48685acb617d259de0db2aac1169ac53"}
Nov 02 12:30:18.406848 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.406819    4464 scope.go:122] "RemoveContainer" containerID="401215d1249073130a7e6f9fb725540430ded5e0460a90931bbc5e3933af256d"
Nov 02 12:30:18.504012 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.499935    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts"]
Nov 02 12:30:18.584771 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.584745    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x"
Nov 02 12:30:18.588218 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.588200    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64"
Nov 02 12:30:18.645744 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.645674    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63708af4-2ca1-4a72-94db-fe035c74534a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63708af4-2ca1-4a72-94db-fe035c74534a-maas-api-service-ca\") pod \"63708af4-2ca1-4a72-94db-fe035c74534a\" (UID: \"63708af4-2ca1-4a72-94db-fe035c74534a\") "
Nov 02 12:30:18.645744 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.645718    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24def35-99de-4046-a464-9758ccc8ab08-kube-api-access-rvt29\" (UniqueName: \"kubernetes.io/projected/c24def35-99de-4046-a464-9758ccc8ab08-kube-api-access-rvt29\") pod \"c24def35-99de-4046-a464-9758ccc8ab08\" (UID: \"c24def35-99de-4046-a464-9758ccc8ab08\") "
Nov 02 12:30:18.645951 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.645920    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63708af4-2ca1-4a72-94db-fe035c74534a-kube-api-access-tbhbh\" (UniqueName: \"kubernetes.io/projected/63708af4-2ca1-4a72-94db-fe035c74534a-kube-api-access-tbhbh\") pod \"63708af4-2ca1-4a72-94db-fe035c74534a\" (UID: \"63708af4-2ca1-4a72-94db-fe035c74534a\") "
Nov 02 12:30:18.646069 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.646046    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63708af4-2ca1-4a72-94db-fe035c74534a-maas-api-service-ca" pod "63708af4-2ca1-4a72-94db-fe035c74534a" (UID: "63708af4-2ca1-4a72-94db-fe035c74534a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:30:18.646127 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.646106    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c24def35-99de-4046-a464-9758ccc8ab08-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c24def35-99de-4046-a464-9758ccc8ab08-maas-api-service-ca\") pod \"c24def35-99de-4046-a464-9758ccc8ab08\" (UID: \"c24def35-99de-4046-a464-9758ccc8ab08\") "
Nov 02 12:30:18.646429 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.646410    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24def35-99de-4046-a464-9758ccc8ab08-maas-api-service-ca" pod "c24def35-99de-4046-a464-9758ccc8ab08" (UID: "c24def35-99de-4046-a464-9758ccc8ab08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:30:18.646528 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.646512    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c24def35-99de-4046-a464-9758ccc8ab08-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:18.646586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.646536    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63708af4-2ca1-4a72-94db-fe035c74534a-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:18.647894 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.647876    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24def35-99de-4046-a464-9758ccc8ab08-kube-api-access-rvt29" pod "c24def35-99de-4046-a464-9758ccc8ab08" (UID: "c24def35-99de-4046-a464-9758ccc8ab08"). InnerVolumeSpecName "kube-api-access-rvt29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:18.647947 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.647923    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63708af4-2ca1-4a72-94db-fe035c74534a-kube-api-access-tbhbh" pod "63708af4-2ca1-4a72-94db-fe035c74534a" (UID: "63708af4-2ca1-4a72-94db-fe035c74534a"). InnerVolumeSpecName "kube-api-access-tbhbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:18.738178 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.738147    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5"]
Nov 02 12:30:18.746266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.746242    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893710-mhnp5"]
Nov 02 12:30:18.747524 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.747506    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvt29\" (UniqueName: \"kubernetes.io/projected/c24def35-99de-4046-a464-9758ccc8ab08-kube-api-access-rvt29\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:18.747571 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:18.747528    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbhbh\" (UniqueName: \"kubernetes.io/projected/63708af4-2ca1-4a72-94db-fe035c74534a-kube-api-access-tbhbh\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:19.405625 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.405593    4464 generic.go:356] "Generic (PLEG): container finished" podID="21f424a5-2715-409d-8f60-f043ed4e10b4" containerID="a41b1353265f315642cc6de9328d758f48685acb617d259de0db2aac1169ac53" exitCode=60
Nov 02 12:30:19.406164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.405668    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j" event={"ID":"21f424a5-2715-409d-8f60-f043ed4e10b4","Type":"ContainerDied","Data":"a41b1353265f315642cc6de9328d758f48685acb617d259de0db2aac1169ac53"}
Nov 02 12:30:19.406164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.405752    4464 scope.go:122] "RemoveContainer" containerID="e41bfdaeb785fde192e7e56f880522aab9df20b467b5e8e55613752d63848eaf"
Nov 02 12:30:19.406164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.406114    4464 scope.go:122] "RemoveContainer" containerID="a41b1353265f315642cc6de9328d758f48685acb617d259de0db2aac1169ac53"
Nov 02 12:30:19.406461 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:19.406433    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j_odh-ai-gateway-infra(21f424a5-2715-409d-8f60-f043ed4e10b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j" podUID="21f424a5-2715-409d-8f60-f043ed4e10b4"
Nov 02 12:30:19.407338 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.407316    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x"
Nov 02 12:30:19.407338 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.407322    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x" event={"ID":"63708af4-2ca1-4a72-94db-fe035c74534a","Type":"ContainerDied","Data":"de146efadbc497e6a7b9505e04e1bace573b9bb411460751bf680e1f5e9b5d00"}
Nov 02 12:30:19.408852 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.408827    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64" event={"ID":"c24def35-99de-4046-a464-9758ccc8ab08","Type":"ContainerDied","Data":"6531fd94a26f6b2cf39d92b9dac1ddae39d58e1c154abb6136f38925ef8e35d3"}
Nov 02 12:30:19.408966 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.408838    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64"
Nov 02 12:30:19.421438 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.421419    4464 scope.go:122] "RemoveContainer" containerID="d934b827319a19ad4266a840628bc141889205ea88c3920a557a289c1614eadf"
Nov 02 12:30:19.431522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.431501    4464 scope.go:122] "RemoveContainer" containerID="dece0834eba7a9523826c683f464dcf3ab615a82753083d1564c5050a8b3c2d0"
Nov 02 12:30:19.572201 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.572172    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts"
Nov 02 12:30:19.582942 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.582912    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2a2936-897c-4f2c-91cd-94b2142ce27b" path="/var/lib/kubelet/pods/0d2a2936-897c-4f2c-91cd-94b2142ce27b/volumes"
Nov 02 12:30:19.583474 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.583454    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6" path="/var/lib/kubelet/pods/cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6/volumes"
Nov 02 12:30:19.636963 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.636927    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x"]
Nov 02 12:30:19.641345 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.641319    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1703c-29893710-kdz2x"]
Nov 02 12:30:19.657491 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.657425    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8199902-070c-4231-b4ca-6d1a4c30403e-kube-api-access-bd9sk\" (UniqueName: \"kubernetes.io/projected/b8199902-070c-4231-b4ca-6d1a4c30403e-kube-api-access-bd9sk\") pod \"b8199902-070c-4231-b4ca-6d1a4c30403e\" (UID: \"b8199902-070c-4231-b4ca-6d1a4c30403e\") "
Nov 02 12:30:19.657491 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.657467    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8199902-070c-4231-b4ca-6d1a4c30403e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8199902-070c-4231-b4ca-6d1a4c30403e-maas-api-service-ca\") pod \"b8199902-070c-4231-b4ca-6d1a4c30403e\" (UID: \"b8199902-070c-4231-b4ca-6d1a4c30403e\") "
Nov 02 12:30:19.657838 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.657815    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8199902-070c-4231-b4ca-6d1a4c30403e-maas-api-service-ca" pod "b8199902-070c-4231-b4ca-6d1a4c30403e" (UID: "b8199902-070c-4231-b4ca-6d1a4c30403e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:30:19.659525 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.659506    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8199902-070c-4231-b4ca-6d1a4c30403e-kube-api-access-bd9sk" pod "b8199902-070c-4231-b4ca-6d1a4c30403e" (UID: "b8199902-070c-4231-b4ca-6d1a4c30403e"). InnerVolumeSpecName "kube-api-access-bd9sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:19.758878 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.758845    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd9sk\" (UniqueName: \"kubernetes.io/projected/b8199902-070c-4231-b4ca-6d1a4c30403e-kube-api-access-bd9sk\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:19.758878 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.758876    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8199902-070c-4231-b4ca-6d1a4c30403e-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:19.868085 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.868051    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64"]
Nov 02 12:30:19.872080 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.872051    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893710-lcv64"]
Nov 02 12:30:19.988062 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:19.988027    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j"]
Nov 02 12:30:20.421195 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:20.421099    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts" event={"ID":"b8199902-070c-4231-b4ca-6d1a4c30403e","Type":"ContainerDied","Data":"9f43185afd58d55a33363fdcf932cb089f3ee29de4ace18363d6a3156ec6bf93"}
Nov 02 12:30:20.421195 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:20.421115    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts"
Nov 02 12:30:20.421725 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:20.421181    4464 scope.go:122] "RemoveContainer" containerID="8c735366e52cdbebf8c1f48a57734500237790d79e22926fd2273e0c49113b98"
Nov 02 12:30:20.565040 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:20.565016    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j"
Nov 02 12:30:20.591341 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:20.591315    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts"]
Nov 02 12:30:20.595330 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:20.595311    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03c4e8-29893710-725ts"]
Nov 02 12:30:20.668924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:20.668900    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f424a5-2715-409d-8f60-f043ed4e10b4-kube-api-access-s9tk4\" (UniqueName: \"kubernetes.io/projected/21f424a5-2715-409d-8f60-f043ed4e10b4-kube-api-access-s9tk4\") pod \"21f424a5-2715-409d-8f60-f043ed4e10b4\" (UID: \"21f424a5-2715-409d-8f60-f043ed4e10b4\") "
Nov 02 12:30:20.669072 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:20.668936    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21f424a5-2715-409d-8f60-f043ed4e10b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f424a5-2715-409d-8f60-f043ed4e10b4-maas-api-service-ca\") pod \"21f424a5-2715-409d-8f60-f043ed4e10b4\" (UID: \"21f424a5-2715-409d-8f60-f043ed4e10b4\") "
Nov 02 12:30:20.669351 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:20.669330    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f424a5-2715-409d-8f60-f043ed4e10b4-maas-api-service-ca" pod "21f424a5-2715-409d-8f60-f043ed4e10b4" (UID: "21f424a5-2715-409d-8f60-f043ed4e10b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:30:20.671047 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:20.671026    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f424a5-2715-409d-8f60-f043ed4e10b4-kube-api-access-s9tk4" pod "21f424a5-2715-409d-8f60-f043ed4e10b4" (UID: "21f424a5-2715-409d-8f60-f043ed4e10b4"). InnerVolumeSpecName "kube-api-access-s9tk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:20.770496 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:20.770466    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9tk4\" (UniqueName: \"kubernetes.io/projected/21f424a5-2715-409d-8f60-f043ed4e10b4-kube-api-access-s9tk4\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:20.770496 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:20.770494    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21f424a5-2715-409d-8f60-f043ed4e10b4-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:21.432867 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:21.432832    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j"
Nov 02 12:30:21.433330 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:21.432862    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j" event={"ID":"21f424a5-2715-409d-8f60-f043ed4e10b4","Type":"ContainerDied","Data":"cc2024ec5f92050bc3107168be29e3a2792c1dc24794898eba0bd8db9cf9b81f"}
Nov 02 12:30:21.433330 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:21.432932    4464 scope.go:122] "RemoveContainer" containerID="a41b1353265f315642cc6de9328d758f48685acb617d259de0db2aac1169ac53"
Nov 02 12:30:21.597288 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:21.596874    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63708af4-2ca1-4a72-94db-fe035c74534a" path="/var/lib/kubelet/pods/63708af4-2ca1-4a72-94db-fe035c74534a/volumes"
Nov 02 12:30:21.602995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:21.597524    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8199902-070c-4231-b4ca-6d1a4c30403e" path="/var/lib/kubelet/pods/b8199902-070c-4231-b4ca-6d1a4c30403e/volumes"
Nov 02 12:30:21.602995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:21.598117    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24def35-99de-4046-a464-9758ccc8ab08" path="/var/lib/kubelet/pods/c24def35-99de-4046-a464-9758ccc8ab08/volumes"
Nov 02 12:30:21.631089 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:21.631018    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j"]
Nov 02 12:30:21.651679 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:21.651636    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-02304760-29893710-5lw4j"]
Nov 02 12:30:22.355003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:22.348110    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544" podUID="155887c3-90e0-45dc-9ae2-22c0469dfef6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:30:23.024263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:23.018028    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"]
Nov 02 12:30:23.024263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:23.020046    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544" podUID="155887c3-90e0-45dc-9ae2-22c0469dfef6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:30:23.056699 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:23.056406    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"]
Nov 02 12:30:23.587761 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:23.587723    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21f424a5-2715-409d-8f60-f043ed4e10b4" path="/var/lib/kubelet/pods/21f424a5-2715-409d-8f60-f043ed4e10b4/volumes"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.446111    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"]
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447117    4464 cpu_manager.go:414] "containerMap: removing container" podUID="cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447139    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447155    4464 cpu_manager.go:414] "containerMap: removing container" podUID="0d2a2936-897c-4f2c-91cd-94b2142ce27b" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447164    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2a2936-897c-4f2c-91cd-94b2142ce27b" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447175    4464 cpu_manager.go:414] "containerMap: removing container" podUID="c24def35-99de-4046-a464-9758ccc8ab08" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447183    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24def35-99de-4046-a464-9758ccc8ab08" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447205    4464 cpu_manager.go:414] "containerMap: removing container" podUID="21f424a5-2715-409d-8f60-f043ed4e10b4" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447213    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21f424a5-2715-409d-8f60-f043ed4e10b4" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447228    4464 cpu_manager.go:414] "containerMap: removing container" podUID="c24def35-99de-4046-a464-9758ccc8ab08" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447237    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24def35-99de-4046-a464-9758ccc8ab08" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447258    4464 cpu_manager.go:414] "containerMap: removing container" podUID="b8199902-070c-4231-b4ca-6d1a4c30403e" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447265    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8199902-070c-4231-b4ca-6d1a4c30403e" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447282    4464 cpu_manager.go:414] "containerMap: removing container" podUID="21f424a5-2715-409d-8f60-f043ed4e10b4" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447290    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21f424a5-2715-409d-8f60-f043ed4e10b4" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447303    4464 cpu_manager.go:414] "containerMap: removing container" podUID="c24def35-99de-4046-a464-9758ccc8ab08" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447310    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24def35-99de-4046-a464-9758ccc8ab08" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447331    4464 cpu_manager.go:414] "containerMap: removing container" podUID="21f424a5-2715-409d-8f60-f043ed4e10b4" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447338    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21f424a5-2715-409d-8f60-f043ed4e10b4" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447347    4464 cpu_manager.go:414] "containerMap: removing container" podUID="63708af4-2ca1-4a72-94db-fe035c74534a" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447354    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63708af4-2ca1-4a72-94db-fe035c74534a" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447369    4464 cpu_manager.go:414] "containerMap: removing container" podUID="0d2a2936-897c-4f2c-91cd-94b2142ce27b" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447376    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2a2936-897c-4f2c-91cd-94b2142ce27b" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447395    4464 cpu_manager.go:414] "containerMap: removing container" podUID="63708af4-2ca1-4a72-94db-fe035c74534a" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447402    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63708af4-2ca1-4a72-94db-fe035c74534a" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447413    4464 cpu_manager.go:414] "containerMap: removing container" podUID="b8199902-070c-4231-b4ca-6d1a4c30403e" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447420    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8199902-070c-4231-b4ca-6d1a4c30403e" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447439    4464 cpu_manager.go:414] "containerMap: removing container" podUID="cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447448    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447585    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="21f424a5-2715-409d-8f60-f043ed4e10b4" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447598    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24def35-99de-4046-a464-9758ccc8ab08" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447610    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8199902-070c-4231-b4ca-6d1a4c30403e" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447619    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="21f424a5-2715-409d-8f60-f043ed4e10b4" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447630    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24def35-99de-4046-a464-9758ccc8ab08" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447644    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="63708af4-2ca1-4a72-94db-fe035c74534a" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447654    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2a2936-897c-4f2c-91cd-94b2142ce27b" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447663    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="63708af4-2ca1-4a72-94db-fe035c74534a" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447672    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447681    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2a2936-897c-4f2c-91cd-94b2142ce27b" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447690    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8199902-070c-4231-b4ca-6d1a4c30403e" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447701    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6" containerName="cleanup"
Nov 02 12:30:25.460999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.447714    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6" containerName="cleanup"
Nov 02 12:30:25.475334 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.475309    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"
Nov 02 12:30:25.525013 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.523146    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"]
Nov 02 12:30:25.562436 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.561295    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:30:25.562436 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.561422    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:30:25.562436 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.561522    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:30:25.562436 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.561553    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:30:25.646024 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.628745    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a7bd74-36f4-4974-975a-506f257b6a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc460cdc-7rbq8\" (UID: \"54a7bd74-36f4-4974-975a-506f257b6a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"
Nov 02 12:30:25.646024 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.628960    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a7bd74-36f4-4974-975a-506f257b6a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc460cdc-7rbq8\" (UID: \"54a7bd74-36f4-4974-975a-506f257b6a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"
Nov 02 12:30:25.646024 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.629056    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph4tx\" (UniqueName: \"kubernetes.io/projected/54a7bd74-36f4-4974-975a-506f257b6a60-kube-api-access-ph4tx\") pod \"maas-api-revoke-sub-fc460cdc-7rbq8\" (UID: \"54a7bd74-36f4-4974-975a-506f257b6a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"
Nov 02 12:30:25.790014 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.732088    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a7bd74-36f4-4974-975a-506f257b6a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc460cdc-7rbq8\" (UID: \"54a7bd74-36f4-4974-975a-506f257b6a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"
Nov 02 12:30:25.795647 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.790446    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mpqgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:30:25.804000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.795939    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm" podUID="d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:30:25.834580 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.804761    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a7bd74-36f4-4974-975a-506f257b6a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc460cdc-7rbq8\" (UID: \"54a7bd74-36f4-4974-975a-506f257b6a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"
Nov 02 12:30:25.834875 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.834852    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph4tx\" (UniqueName: \"kubernetes.io/projected/54a7bd74-36f4-4974-975a-506f257b6a60-kube-api-access-ph4tx\") pod \"maas-api-revoke-sub-fc460cdc-7rbq8\" (UID: \"54a7bd74-36f4-4974-975a-506f257b6a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"
Nov 02 12:30:25.835182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.833910    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a7bd74-36f4-4974-975a-506f257b6a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc460cdc-7rbq8\" (UID: \"54a7bd74-36f4-4974-975a-506f257b6a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"
Nov 02 12:30:25.950001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:25.933490    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a7bd74-36f4-4974-975a-506f257b6a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc460cdc-7rbq8\" (UID: \"54a7bd74-36f4-4974-975a-506f257b6a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"
Nov 02 12:30:26.172062 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:26.158636    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph4tx\" (UniqueName: \"kubernetes.io/projected/54a7bd74-36f4-4974-975a-506f257b6a60-kube-api-access-ph4tx\") pod \"maas-api-revoke-sub-fc460cdc-7rbq8\" (UID: \"54a7bd74-36f4-4974-975a-506f257b6a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"
Nov 02 12:30:26.440345 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:26.440247    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"
Nov 02 12:30:27.557257 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:27.557212    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"]
Nov 02 12:30:27.613711 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:27.613679    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8" event={"ID":"54a7bd74-36f4-4974-975a-506f257b6a60","Type":"ContainerStarted","Data":"8c5bc1073376ac5f73f093a1f59e267ef163615baff5de1055fa04780fabac1e"}
Nov 02 12:30:28.019247 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.019210    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544" podUID="155887c3-90e0-45dc-9ae2-22c0469dfef6" containerName="payload-processing" containerID="cri-o://4ca954cbb2a7fc6227daa3adb3da9bb256b388c8f23d485af73b9821a6265850" gracePeriod=25
Nov 02 12:30:28.057261 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.057208    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm" podUID="1068cf8a-7fc4-48ea-9522-3fc6e914891a" containerName="payload-processing" containerID="cri-o://b312bf4d7aa60ff6f73723af08c3aa7c1dd28cd97e3385fe01d233e159d6a1cb" gracePeriod=25
Nov 02 12:30:28.303719 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:28.303689    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:30:28.306929 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.306906    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"
Nov 02 12:30:28.335838 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.335815    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:30:28.403082 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.403050    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1068cf8a-7fc4-48ea-9522-3fc6e914891a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1068cf8a-7fc4-48ea-9522-3fc6e914891a-plugins-config-volume\") pod \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\" (UID: \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\") "
Nov 02 12:30:28.403251 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.403119    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/155887c3-90e0-45dc-9ae2-22c0469dfef6-kube-api-access-j5jbx\" (UniqueName: \"kubernetes.io/projected/155887c3-90e0-45dc-9ae2-22c0469dfef6-kube-api-access-j5jbx\") pod \"155887c3-90e0-45dc-9ae2-22c0469dfef6\" (UID: \"155887c3-90e0-45dc-9ae2-22c0469dfef6\") "
Nov 02 12:30:28.403251 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.403179    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1068cf8a-7fc4-48ea-9522-3fc6e914891a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1068cf8a-7fc4-48ea-9522-3fc6e914891a-provider-credentials\") pod \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\" (UID: \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\") "
Nov 02 12:30:28.403251 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.403200    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1068cf8a-7fc4-48ea-9522-3fc6e914891a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1068cf8a-7fc4-48ea-9522-3fc6e914891a-routing-overlay-volume\") pod \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\" (UID: \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\") "
Nov 02 12:30:28.403251 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.403219    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/155887c3-90e0-45dc-9ae2-22c0469dfef6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/155887c3-90e0-45dc-9ae2-22c0469dfef6-provider-credentials\") pod \"155887c3-90e0-45dc-9ae2-22c0469dfef6\" (UID: \"155887c3-90e0-45dc-9ae2-22c0469dfef6\") "
Nov 02 12:30:28.403251 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.403247    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/155887c3-90e0-45dc-9ae2-22c0469dfef6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/155887c3-90e0-45dc-9ae2-22c0469dfef6-routing-overlay-volume\") pod \"155887c3-90e0-45dc-9ae2-22c0469dfef6\" (UID: \"155887c3-90e0-45dc-9ae2-22c0469dfef6\") "
Nov 02 12:30:28.403540 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.403277    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/155887c3-90e0-45dc-9ae2-22c0469dfef6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/155887c3-90e0-45dc-9ae2-22c0469dfef6-plugins-config-volume\") pod \"155887c3-90e0-45dc-9ae2-22c0469dfef6\" (UID: \"155887c3-90e0-45dc-9ae2-22c0469dfef6\") "
Nov 02 12:30:28.403540 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.403308    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1068cf8a-7fc4-48ea-9522-3fc6e914891a-kube-api-access-xbzn8\" (UniqueName: \"kubernetes.io/projected/1068cf8a-7fc4-48ea-9522-3fc6e914891a-kube-api-access-xbzn8\") pod \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\" (UID: \"1068cf8a-7fc4-48ea-9522-3fc6e914891a\") "
Nov 02 12:30:28.403540 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.403493    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1068cf8a-7fc4-48ea-9522-3fc6e914891a-plugins-config-volume" pod "1068cf8a-7fc4-48ea-9522-3fc6e914891a" (UID: "1068cf8a-7fc4-48ea-9522-3fc6e914891a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:30:28.403714 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.403631    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/155887c3-90e0-45dc-9ae2-22c0469dfef6-routing-overlay-volume" pod "155887c3-90e0-45dc-9ae2-22c0469dfef6" (UID: "155887c3-90e0-45dc-9ae2-22c0469dfef6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:30:28.403779 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.403722    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/155887c3-90e0-45dc-9ae2-22c0469dfef6-plugins-config-volume" pod "155887c3-90e0-45dc-9ae2-22c0469dfef6" (UID: "155887c3-90e0-45dc-9ae2-22c0469dfef6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:30:28.404035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.404007    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1068cf8a-7fc4-48ea-9522-3fc6e914891a-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:28.404035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.404031    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1068cf8a-7fc4-48ea-9522-3fc6e914891a-routing-overlay-volume" pod "1068cf8a-7fc4-48ea-9522-3fc6e914891a" (UID: "1068cf8a-7fc4-48ea-9522-3fc6e914891a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:30:28.404245 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.404038    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/155887c3-90e0-45dc-9ae2-22c0469dfef6-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:28.404245 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.404065    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/155887c3-90e0-45dc-9ae2-22c0469dfef6-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:28.405353 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.405332    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1068cf8a-7fc4-48ea-9522-3fc6e914891a-provider-credentials" pod "1068cf8a-7fc4-48ea-9522-3fc6e914891a" (UID: "1068cf8a-7fc4-48ea-9522-3fc6e914891a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:28.405453 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.405375    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155887c3-90e0-45dc-9ae2-22c0469dfef6-kube-api-access-j5jbx" pod "155887c3-90e0-45dc-9ae2-22c0469dfef6" (UID: "155887c3-90e0-45dc-9ae2-22c0469dfef6"). InnerVolumeSpecName "kube-api-access-j5jbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:28.405712 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.405695    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155887c3-90e0-45dc-9ae2-22c0469dfef6-provider-credentials" pod "155887c3-90e0-45dc-9ae2-22c0469dfef6" (UID: "155887c3-90e0-45dc-9ae2-22c0469dfef6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:28.405758 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.405729    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1068cf8a-7fc4-48ea-9522-3fc6e914891a-kube-api-access-xbzn8" pod "1068cf8a-7fc4-48ea-9522-3fc6e914891a" (UID: "1068cf8a-7fc4-48ea-9522-3fc6e914891a"). InnerVolumeSpecName "kube-api-access-xbzn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:28.505055 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.505030    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5jbx\" (UniqueName: \"kubernetes.io/projected/155887c3-90e0-45dc-9ae2-22c0469dfef6-kube-api-access-j5jbx\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:28.505055 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.505054    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1068cf8a-7fc4-48ea-9522-3fc6e914891a-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:28.505221 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.505066    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1068cf8a-7fc4-48ea-9522-3fc6e914891a-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:28.505221 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.505075    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/155887c3-90e0-45dc-9ae2-22c0469dfef6-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:28.505221 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.505084    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbzn8\" (UniqueName: \"kubernetes.io/projected/1068cf8a-7fc4-48ea-9522-3fc6e914891a-kube-api-access-xbzn8\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:28.621369 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.621337    4464 generic.go:356] "Generic (PLEG): container finished" podID="1068cf8a-7fc4-48ea-9522-3fc6e914891a" containerID="b312bf4d7aa60ff6f73723af08c3aa7c1dd28cd97e3385fe01d233e159d6a1cb" exitCode=0
Nov 02 12:30:28.621751 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.621379    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"
Nov 02 12:30:28.621751 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.621425    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm" event={"ID":"1068cf8a-7fc4-48ea-9522-3fc6e914891a","Type":"ContainerDied","Data":"b312bf4d7aa60ff6f73723af08c3aa7c1dd28cd97e3385fe01d233e159d6a1cb"}
Nov 02 12:30:28.621751 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.621476    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm" event={"ID":"1068cf8a-7fc4-48ea-9522-3fc6e914891a","Type":"ContainerDied","Data":"fecd13ded6c399cf705f6230e6aa7ec4f0d1cade96bd27db681ec791ed7f1ef2"}
Nov 02 12:30:28.621751 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.621505    4464 scope.go:122] "RemoveContainer" containerID="b312bf4d7aa60ff6f73723af08c3aa7c1dd28cd97e3385fe01d233e159d6a1cb"
Nov 02 12:30:28.622850 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.622830    4464 generic.go:356] "Generic (PLEG): container finished" podID="155887c3-90e0-45dc-9ae2-22c0469dfef6" containerID="4ca954cbb2a7fc6227daa3adb3da9bb256b388c8f23d485af73b9821a6265850" exitCode=0
Nov 02 12:30:28.622931 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.622893    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"
Nov 02 12:30:28.622931 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.622913    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544" event={"ID":"155887c3-90e0-45dc-9ae2-22c0469dfef6","Type":"ContainerDied","Data":"4ca954cbb2a7fc6227daa3adb3da9bb256b388c8f23d485af73b9821a6265850"}
Nov 02 12:30:28.623047 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.622962    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bc9c479f4-df544" event={"ID":"155887c3-90e0-45dc-9ae2-22c0469dfef6","Type":"ContainerDied","Data":"60e4d36661546f01a0fd8b684653768ff4f5943e47eeaee646413ca330de9034"}
Nov 02 12:30:28.624686 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.624666    4464 generic.go:356] "Generic (PLEG): container finished" podID="54a7bd74-36f4-4974-975a-506f257b6a60" containerID="75d0bd1b036e1be961e5e52f9eabdb94a9221d26b843d8757935567f7b238c92" exitCode=0
Nov 02 12:30:28.624769 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.624693    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8" event={"ID":"54a7bd74-36f4-4974-975a-506f257b6a60","Type":"ContainerDied","Data":"75d0bd1b036e1be961e5e52f9eabdb94a9221d26b843d8757935567f7b238c92"}
Nov 02 12:30:28.631930 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.631909    4464 scope.go:122] "RemoveContainer" containerID="b312bf4d7aa60ff6f73723af08c3aa7c1dd28cd97e3385fe01d233e159d6a1cb"
Nov 02 12:30:28.632239 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:28.632220    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b312bf4d7aa60ff6f73723af08c3aa7c1dd28cd97e3385fe01d233e159d6a1cb\": container with ID starting with b312bf4d7aa60ff6f73723af08c3aa7c1dd28cd97e3385fe01d233e159d6a1cb not found: ID does not exist" containerID="b312bf4d7aa60ff6f73723af08c3aa7c1dd28cd97e3385fe01d233e159d6a1cb"
Nov 02 12:30:28.632304 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.632246    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b312bf4d7aa60ff6f73723af08c3aa7c1dd28cd97e3385fe01d233e159d6a1cb"} err="failed to get container status \"b312bf4d7aa60ff6f73723af08c3aa7c1dd28cd97e3385fe01d233e159d6a1cb\": rpc error: code = NotFound desc = could not find container \"b312bf4d7aa60ff6f73723af08c3aa7c1dd28cd97e3385fe01d233e159d6a1cb\": container with ID starting with b312bf4d7aa60ff6f73723af08c3aa7c1dd28cd97e3385fe01d233e159d6a1cb not found: ID does not exist"
Nov 02 12:30:28.632304 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.632263    4464 scope.go:122] "RemoveContainer" containerID="4ca954cbb2a7fc6227daa3adb3da9bb256b388c8f23d485af73b9821a6265850"
Nov 02 12:30:28.640957 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.640941    4464 scope.go:122] "RemoveContainer" containerID="4ca954cbb2a7fc6227daa3adb3da9bb256b388c8f23d485af73b9821a6265850"
Nov 02 12:30:28.641259 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:28.641238    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca954cbb2a7fc6227daa3adb3da9bb256b388c8f23d485af73b9821a6265850\": container with ID starting with 4ca954cbb2a7fc6227daa3adb3da9bb256b388c8f23d485af73b9821a6265850 not found: ID does not exist" containerID="4ca954cbb2a7fc6227daa3adb3da9bb256b388c8f23d485af73b9821a6265850"
Nov 02 12:30:28.641345 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.641263    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca954cbb2a7fc6227daa3adb3da9bb256b388c8f23d485af73b9821a6265850"} err="failed to get container status \"4ca954cbb2a7fc6227daa3adb3da9bb256b388c8f23d485af73b9821a6265850\": rpc error: code = NotFound desc = could not find container \"4ca954cbb2a7fc6227daa3adb3da9bb256b388c8f23d485af73b9821a6265850\": container with ID starting with 4ca954cbb2a7fc6227daa3adb3da9bb256b388c8f23d485af73b9821a6265850 not found: ID does not exist"
Nov 02 12:30:28.907595 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.907564    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"]
Nov 02 12:30:28.911681 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:28.911655    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cc9844f6-pv7vm"]
Nov 02 12:30:29.022454 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:29.022422    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"]
Nov 02 12:30:29.028460 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:29.028435    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bc9c479f4-df544"]
Nov 02 12:30:29.143182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:29.143150    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"]
Nov 02 12:30:29.582573 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:29.582533    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1068cf8a-7fc4-48ea-9522-3fc6e914891a" path="/var/lib/kubelet/pods/1068cf8a-7fc4-48ea-9522-3fc6e914891a/volumes"
Nov 02 12:30:29.582999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:29.582961    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="155887c3-90e0-45dc-9ae2-22c0469dfef6" path="/var/lib/kubelet/pods/155887c3-90e0-45dc-9ae2-22c0469dfef6/volumes"
Nov 02 12:30:29.781545 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:29.781525    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"
Nov 02 12:30:29.815150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:29.815126    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a7bd74-36f4-4974-975a-506f257b6a60-kube-api-access-ph4tx\" (UniqueName: \"kubernetes.io/projected/54a7bd74-36f4-4974-975a-506f257b6a60-kube-api-access-ph4tx\") pod \"54a7bd74-36f4-4974-975a-506f257b6a60\" (UID: \"54a7bd74-36f4-4974-975a-506f257b6a60\") "
Nov 02 12:30:29.815276 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:29.815247    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a7bd74-36f4-4974-975a-506f257b6a60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a7bd74-36f4-4974-975a-506f257b6a60-maas-api-openshift-ca\") pod \"54a7bd74-36f4-4974-975a-506f257b6a60\" (UID: \"54a7bd74-36f4-4974-975a-506f257b6a60\") "
Nov 02 12:30:29.815321 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:29.815310    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54a7bd74-36f4-4974-975a-506f257b6a60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a7bd74-36f4-4974-975a-506f257b6a60-maas-api-xks-ca\") pod \"54a7bd74-36f4-4974-975a-506f257b6a60\" (UID: \"54a7bd74-36f4-4974-975a-506f257b6a60\") "
Nov 02 12:30:29.815600 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:29.815572    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a7bd74-36f4-4974-975a-506f257b6a60-maas-api-openshift-ca" pod "54a7bd74-36f4-4974-975a-506f257b6a60" (UID: "54a7bd74-36f4-4974-975a-506f257b6a60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:30:29.815706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:29.815686    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a7bd74-36f4-4974-975a-506f257b6a60-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:29.817039 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:29.817013    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a7bd74-36f4-4974-975a-506f257b6a60-kube-api-access-ph4tx" pod "54a7bd74-36f4-4974-975a-506f257b6a60" (UID: "54a7bd74-36f4-4974-975a-506f257b6a60"). InnerVolumeSpecName "kube-api-access-ph4tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:29.817148 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:29.817090    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a7bd74-36f4-4974-975a-506f257b6a60-maas-api-xks-ca" pod "54a7bd74-36f4-4974-975a-506f257b6a60" (UID: "54a7bd74-36f4-4974-975a-506f257b6a60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:30:29.916831 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:29.916750    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a7bd74-36f4-4974-975a-506f257b6a60-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:29.916831 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:29.916786    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph4tx\" (UniqueName: \"kubernetes.io/projected/54a7bd74-36f4-4974-975a-506f257b6a60-kube-api-access-ph4tx\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:30.642428 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:30.642401    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"
Nov 02 12:30:30.642616 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:30.642422    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8" event={"ID":"54a7bd74-36f4-4974-975a-506f257b6a60","Type":"ContainerDied","Data":"8c5bc1073376ac5f73f093a1f59e267ef163615baff5de1055fa04780fabac1e"}
Nov 02 12:30:30.642616 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:30.642467    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c5bc1073376ac5f73f093a1f59e267ef163615baff5de1055fa04780fabac1e"
Nov 02 12:30:34.295176 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:34.295140    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:30:41.809628 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.809597    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"]
Nov 02 12:30:41.810207 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810193    4464 cpu_manager.go:414] "containerMap: removing container" podUID="cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6" containerName="cleanup"
Nov 02 12:30:41.810266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810207    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd70f0b2-a4fd-49cf-9ac0-6a587d4ad9b6" containerName="cleanup"
Nov 02 12:30:41.810266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810225    4464 cpu_manager.go:414] "containerMap: removing container" podUID="155887c3-90e0-45dc-9ae2-22c0469dfef6" containerName="payload-processing"
Nov 02 12:30:41.810266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810231    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="155887c3-90e0-45dc-9ae2-22c0469dfef6" containerName="payload-processing"
Nov 02 12:30:41.810266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810242    4464 cpu_manager.go:414] "containerMap: removing container" podUID="54a7bd74-36f4-4974-975a-506f257b6a60" containerName="revoke-keys"
Nov 02 12:30:41.810266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810247    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a7bd74-36f4-4974-975a-506f257b6a60" containerName="revoke-keys"
Nov 02 12:30:41.810266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810265    4464 cpu_manager.go:414] "containerMap: removing container" podUID="0d2a2936-897c-4f2c-91cd-94b2142ce27b" containerName="cleanup"
Nov 02 12:30:41.810451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810270    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2a2936-897c-4f2c-91cd-94b2142ce27b" containerName="cleanup"
Nov 02 12:30:41.810451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810278    4464 cpu_manager.go:414] "containerMap: removing container" podUID="1068cf8a-7fc4-48ea-9522-3fc6e914891a" containerName="payload-processing"
Nov 02 12:30:41.810451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810282    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1068cf8a-7fc4-48ea-9522-3fc6e914891a" containerName="payload-processing"
Nov 02 12:30:41.810451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810293    4464 cpu_manager.go:414] "containerMap: removing container" podUID="63708af4-2ca1-4a72-94db-fe035c74534a" containerName="cleanup"
Nov 02 12:30:41.810451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810297    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63708af4-2ca1-4a72-94db-fe035c74534a" containerName="cleanup"
Nov 02 12:30:41.810451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810303    4464 cpu_manager.go:414] "containerMap: removing container" podUID="b8199902-070c-4231-b4ca-6d1a4c30403e" containerName="cleanup"
Nov 02 12:30:41.810451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810308    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8199902-070c-4231-b4ca-6d1a4c30403e" containerName="cleanup"
Nov 02 12:30:41.810451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810387    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="63708af4-2ca1-4a72-94db-fe035c74534a" containerName="cleanup"
Nov 02 12:30:41.810451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810396    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="1068cf8a-7fc4-48ea-9522-3fc6e914891a" containerName="payload-processing"
Nov 02 12:30:41.810451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810403    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2a2936-897c-4f2c-91cd-94b2142ce27b" containerName="cleanup"
Nov 02 12:30:41.810451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810415    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="155887c3-90e0-45dc-9ae2-22c0469dfef6" containerName="payload-processing"
Nov 02 12:30:41.810451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810421    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8199902-070c-4231-b4ca-6d1a4c30403e" containerName="cleanup"
Nov 02 12:30:41.810451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810425    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a7bd74-36f4-4974-975a-506f257b6a60" containerName="revoke-keys"
Nov 02 12:30:41.810451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810431    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="21f424a5-2715-409d-8f60-f043ed4e10b4" containerName="cleanup"
Nov 02 12:30:41.810451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.810437    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24def35-99de-4046-a464-9758ccc8ab08" containerName="cleanup"
Nov 02 12:30:41.813923 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.813907    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"
Nov 02 12:30:41.870211 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.870182    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"]
Nov 02 12:30:41.888695 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.888665    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzjcx\" (UniqueName: \"kubernetes.io/projected/75611cca-49e2-4172-8d37-089b6c0295dd-kube-api-access-fzjcx\") pod \"maas-api-revoke-sub-7a6e55f8-87lt7\" (UID: \"75611cca-49e2-4172-8d37-089b6c0295dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"
Nov 02 12:30:41.888920 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.888897    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75611cca-49e2-4172-8d37-089b6c0295dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a6e55f8-87lt7\" (UID: \"75611cca-49e2-4172-8d37-089b6c0295dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"
Nov 02 12:30:41.889012 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.888938    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75611cca-49e2-4172-8d37-089b6c0295dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a6e55f8-87lt7\" (UID: \"75611cca-49e2-4172-8d37-089b6c0295dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"
Nov 02 12:30:41.916817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.916788    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:30:41.916943 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.916827    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:30:41.916943 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.916855    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:30:41.917134 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.917122    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:30:41.990498 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.990471    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75611cca-49e2-4172-8d37-089b6c0295dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a6e55f8-87lt7\" (UID: \"75611cca-49e2-4172-8d37-089b6c0295dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"
Nov 02 12:30:41.990636 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.990503    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75611cca-49e2-4172-8d37-089b6c0295dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a6e55f8-87lt7\" (UID: \"75611cca-49e2-4172-8d37-089b6c0295dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"
Nov 02 12:30:41.990636 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.990574    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzjcx\" (UniqueName: \"kubernetes.io/projected/75611cca-49e2-4172-8d37-089b6c0295dd-kube-api-access-fzjcx\") pod \"maas-api-revoke-sub-7a6e55f8-87lt7\" (UID: \"75611cca-49e2-4172-8d37-089b6c0295dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"
Nov 02 12:30:41.991284 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.991248    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75611cca-49e2-4172-8d37-089b6c0295dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a6e55f8-87lt7\" (UID: \"75611cca-49e2-4172-8d37-089b6c0295dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"
Nov 02 12:30:41.992874 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:41.992856    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75611cca-49e2-4172-8d37-089b6c0295dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a6e55f8-87lt7\" (UID: \"75611cca-49e2-4172-8d37-089b6c0295dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"
Nov 02 12:30:42.052174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:42.052147    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzjcx\" (UniqueName: \"kubernetes.io/projected/75611cca-49e2-4172-8d37-089b6c0295dd-kube-api-access-fzjcx\") pod \"maas-api-revoke-sub-7a6e55f8-87lt7\" (UID: \"75611cca-49e2-4172-8d37-089b6c0295dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"
Nov 02 12:30:42.127500 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:42.127439    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"
Nov 02 12:30:42.505024 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:42.504997    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"]
Nov 02 12:30:42.505812 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:30:42.505787    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75611cca_49e2_4172_8d37_089b6c0295dd.slice/crio-e4d3594802ee2cde7f5c449f3e8e926dd6a5e4561f283b4000e39003306035c4 WatchSource:0}: Error finding container e4d3594802ee2cde7f5c449f3e8e926dd6a5e4561f283b4000e39003306035c4: Status 404 returned error can't find the container with id e4d3594802ee2cde7f5c449f3e8e926dd6a5e4561f283b4000e39003306035c4
Nov 02 12:30:42.766403 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:42.766371    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7" event={"ID":"75611cca-49e2-4172-8d37-089b6c0295dd","Type":"ContainerStarted","Data":"e4d3594802ee2cde7f5c449f3e8e926dd6a5e4561f283b4000e39003306035c4"}
Nov 02 12:30:43.259677 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:43.259641    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"]
Nov 02 12:30:43.310200 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:43.310174    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"]
Nov 02 12:30:43.703045 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:43.702512    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"]
Nov 02 12:30:43.857164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:43.857131    4464 generic.go:356] "Generic (PLEG): container finished" podID="75611cca-49e2-4172-8d37-089b6c0295dd" containerID="07d2b813b2d4b743a5cc5e6393043b00bbf4e5c9685314a8f4f96c084af52516" exitCode=0
Nov 02 12:30:43.857344 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:43.857215    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7" event={"ID":"75611cca-49e2-4172-8d37-089b6c0295dd","Type":"ContainerDied","Data":"07d2b813b2d4b743a5cc5e6393043b00bbf4e5c9685314a8f4f96c084af52516"}
Nov 02 12:30:44.162984 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.162945    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj" podUID="d30985d1-0ddd-486f-8104-94f3bdb4c049" containerName="main" containerID="cri-o://59268ddc007b5d35d67aa7331714851578cdaf3ee418a3792af697c1d2686464" gracePeriod=45
Nov 02 12:30:44.428057 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.427238    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:30:44.522207 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.522180    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-home\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-home\") pod \"d30985d1-0ddd-486f-8104-94f3bdb4c049\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") "
Nov 02 12:30:44.522357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.522215    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d30985d1-0ddd-486f-8104-94f3bdb4c049-tls-certs\" (UniqueName: \"kubernetes.io/secret/d30985d1-0ddd-486f-8104-94f3bdb4c049-tls-certs\") pod \"d30985d1-0ddd-486f-8104-94f3bdb4c049\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") "
Nov 02 12:30:44.522357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.522319    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-tmp-dir\") pod \"d30985d1-0ddd-486f-8104-94f3bdb4c049\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") "
Nov 02 12:30:44.522357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.522355    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-model-cache\") pod \"d30985d1-0ddd-486f-8104-94f3bdb4c049\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") "
Nov 02 12:30:44.522496 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.522407    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30985d1-0ddd-486f-8104-94f3bdb4c049-kube-api-access-v4rc5\" (UniqueName: \"kubernetes.io/projected/d30985d1-0ddd-486f-8104-94f3bdb4c049-kube-api-access-v4rc5\") pod \"d30985d1-0ddd-486f-8104-94f3bdb4c049\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") "
Nov 02 12:30:44.522496 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.522430    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-dshm\") pod \"d30985d1-0ddd-486f-8104-94f3bdb4c049\" (UID: \"d30985d1-0ddd-486f-8104-94f3bdb4c049\") "
Nov 02 12:30:44.522496 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.522435    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-home" pod "d30985d1-0ddd-486f-8104-94f3bdb4c049" (UID: "d30985d1-0ddd-486f-8104-94f3bdb4c049"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:44.522655 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.522614    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-tmp-dir" pod "d30985d1-0ddd-486f-8104-94f3bdb4c049" (UID: "d30985d1-0ddd-486f-8104-94f3bdb4c049"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:44.522655 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.522628    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-model-cache" pod "d30985d1-0ddd-486f-8104-94f3bdb4c049" (UID: "d30985d1-0ddd-486f-8104-94f3bdb4c049"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:44.522948 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.522929    4464 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-home\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:44.522948 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.522949    4464 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-tmp-dir\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:44.523132 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.522959    4464 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-model-cache\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:44.524523 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.524498    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-dshm" pod "d30985d1-0ddd-486f-8104-94f3bdb4c049" (UID: "d30985d1-0ddd-486f-8104-94f3bdb4c049"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:44.524523 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.524507    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d30985d1-0ddd-486f-8104-94f3bdb4c049-tls-certs" pod "d30985d1-0ddd-486f-8104-94f3bdb4c049" (UID: "d30985d1-0ddd-486f-8104-94f3bdb4c049"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:30:44.524677 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.524538    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30985d1-0ddd-486f-8104-94f3bdb4c049-kube-api-access-v4rc5" pod "d30985d1-0ddd-486f-8104-94f3bdb4c049" (UID: "d30985d1-0ddd-486f-8104-94f3bdb4c049"). InnerVolumeSpecName "kube-api-access-v4rc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:44.624226 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.624197    4464 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d30985d1-0ddd-486f-8104-94f3bdb4c049-tls-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:44.624426 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.624397    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4rc5\" (UniqueName: \"kubernetes.io/projected/d30985d1-0ddd-486f-8104-94f3bdb4c049-kube-api-access-v4rc5\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:44.624426 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.624420    4464 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d30985d1-0ddd-486f-8104-94f3bdb4c049-dshm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:44.816518 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.816486    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"]
Nov 02 12:30:44.817108 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.817093    4464 cpu_manager.go:414] "containerMap: removing container" podUID="d30985d1-0ddd-486f-8104-94f3bdb4c049" containerName="main"
Nov 02 12:30:44.817166 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.817110    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30985d1-0ddd-486f-8104-94f3bdb4c049" containerName="main"
Nov 02 12:30:44.817230 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.817221    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30985d1-0ddd-486f-8104-94f3bdb4c049" containerName="main"
Nov 02 12:30:44.820826 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.820810    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:44.866085 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.866052    4464 generic.go:356] "Generic (PLEG): container finished" podID="d30985d1-0ddd-486f-8104-94f3bdb4c049" containerID="59268ddc007b5d35d67aa7331714851578cdaf3ee418a3792af697c1d2686464" exitCode=0
Nov 02 12:30:44.866249 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.866109    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"
Nov 02 12:30:44.866249 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.866138    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj" event={"ID":"d30985d1-0ddd-486f-8104-94f3bdb4c049","Type":"ContainerDied","Data":"59268ddc007b5d35d67aa7331714851578cdaf3ee418a3792af697c1d2686464"}
Nov 02 12:30:44.866249 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.866190    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj" event={"ID":"d30985d1-0ddd-486f-8104-94f3bdb4c049","Type":"ContainerDied","Data":"17b2170db13cf723d0ef82ed6bf7541a6e0babdded1f6df2159ec5e4e96d0718"}
Nov 02 12:30:44.866249 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.866213    4464 scope.go:122] "RemoveContainer" containerID="59268ddc007b5d35d67aa7331714851578cdaf3ee418a3792af697c1d2686464"
Nov 02 12:30:44.880958 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.880937    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-cf0095-openshift-default-dockercfg-dfgk5\""
Nov 02 12:30:44.881473 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.881449    4464 scope.go:122] "RemoveContainer" containerID="59268ddc007b5d35d67aa7331714851578cdaf3ee418a3792af697c1d2686464"
Nov 02 12:30:44.881768 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:44.881750    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59268ddc007b5d35d67aa7331714851578cdaf3ee418a3792af697c1d2686464\": container with ID starting with 59268ddc007b5d35d67aa7331714851578cdaf3ee418a3792af697c1d2686464 not found: ID does not exist" containerID="59268ddc007b5d35d67aa7331714851578cdaf3ee418a3792af697c1d2686464"
Nov 02 12:30:44.881838 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.881781    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59268ddc007b5d35d67aa7331714851578cdaf3ee418a3792af697c1d2686464"} err="failed to get container status \"59268ddc007b5d35d67aa7331714851578cdaf3ee418a3792af697c1d2686464\": rpc error: code = NotFound desc = could not find container \"59268ddc007b5d35d67aa7331714851578cdaf3ee418a3792af697c1d2686464\": container with ID starting with 59268ddc007b5d35d67aa7331714851578cdaf3ee418a3792af697c1d2686464 not found: ID does not exist"
Nov 02 12:30:44.885045 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.884964    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"]
Nov 02 12:30:44.928686 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.928644    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a7742a-f60b-4270-8992-2855b79bed42-istio-token\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:44.928865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.928696    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsrmz\" (UniqueName: \"kubernetes.io/projected/a7a7742a-f60b-4270-8992-2855b79bed42-kube-api-access-zsrmz\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:44.928865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.928839    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-workload-certs\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:44.929016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.928887    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-istio-data\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:44.929016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.928941    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-credential-socket\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:44.929016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.929005    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a7742a-f60b-4270-8992-2855b79bed42-istiod-ca-cert\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:44.929199 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.929052    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-workload-socket\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:44.929199 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.929082    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-istio-envoy\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:44.929199 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:44.929115    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a7742a-f60b-4270-8992-2855b79bed42-istio-podinfo\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.112094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.073685    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a7742a-f60b-4270-8992-2855b79bed42-istiod-ca-cert\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.112094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.073755    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-workload-socket\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.112094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.073782    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-istio-envoy\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.112094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.073812    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a7742a-f60b-4270-8992-2855b79bed42-istio-podinfo\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.112094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.073971    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a7742a-f60b-4270-8992-2855b79bed42-istio-token\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.112094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.074019    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsrmz\" (UniqueName: \"kubernetes.io/projected/a7a7742a-f60b-4270-8992-2855b79bed42-kube-api-access-zsrmz\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.112094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.074090    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-workload-certs\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.112094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.074126    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-istio-data\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.112094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.074176    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-credential-socket\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.112094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.074622    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-credential-socket\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.112094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.075576    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a7742a-f60b-4270-8992-2855b79bed42-istiod-ca-cert\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.112094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.075815    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-workload-socket\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.112094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.110625    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a7742a-f60b-4270-8992-2855b79bed42-istio-podinfo\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.112094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.110915    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-workload-certs\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.112094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.111189    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-istio-data\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.113014 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.112834    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"]
Nov 02 12:30:45.141556 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.141350    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/premium-w0-b1703c-kserve-7988b5bcd6-8lgcj"]
Nov 02 12:30:45.142848 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.142815    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-istio-envoy\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.167021 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.164590    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsrmz\" (UniqueName: \"kubernetes.io/projected/a7a7742a-f60b-4270-8992-2855b79bed42-kube-api-access-zsrmz\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.174919 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.174887    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a7742a-f60b-4270-8992-2855b79bed42-istio-token\") pod \"e2e-autores-cf0095-openshift-default-66557c8649-rjm2j\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") " pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.208281 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.208257    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"
Nov 02 12:30:45.276505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.276474    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75611cca-49e2-4172-8d37-089b6c0295dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75611cca-49e2-4172-8d37-089b6c0295dd-maas-api-openshift-ca\") pod \"75611cca-49e2-4172-8d37-089b6c0295dd\" (UID: \"75611cca-49e2-4172-8d37-089b6c0295dd\") "
Nov 02 12:30:45.276659 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.276537    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75611cca-49e2-4172-8d37-089b6c0295dd-kube-api-access-fzjcx\" (UniqueName: \"kubernetes.io/projected/75611cca-49e2-4172-8d37-089b6c0295dd-kube-api-access-fzjcx\") pod \"75611cca-49e2-4172-8d37-089b6c0295dd\" (UID: \"75611cca-49e2-4172-8d37-089b6c0295dd\") "
Nov 02 12:30:45.276812 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.276791    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75611cca-49e2-4172-8d37-089b6c0295dd-maas-api-openshift-ca" pod "75611cca-49e2-4172-8d37-089b6c0295dd" (UID: "75611cca-49e2-4172-8d37-089b6c0295dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:30:45.277070 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.277053    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75611cca-49e2-4172-8d37-089b6c0295dd-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:45.278613 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.278594    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75611cca-49e2-4172-8d37-089b6c0295dd-kube-api-access-fzjcx" pod "75611cca-49e2-4172-8d37-089b6c0295dd" (UID: "75611cca-49e2-4172-8d37-089b6c0295dd"). InnerVolumeSpecName "kube-api-access-fzjcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:45.396800 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.396715    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75611cca-49e2-4172-8d37-089b6c0295dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75611cca-49e2-4172-8d37-089b6c0295dd-maas-api-xks-ca\") pod \"75611cca-49e2-4172-8d37-089b6c0295dd\" (UID: \"75611cca-49e2-4172-8d37-089b6c0295dd\") "
Nov 02 12:30:45.397571 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.397527    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzjcx\" (UniqueName: \"kubernetes.io/projected/75611cca-49e2-4172-8d37-089b6c0295dd-kube-api-access-fzjcx\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:45.406273 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.406229    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75611cca-49e2-4172-8d37-089b6c0295dd-maas-api-xks-ca" pod "75611cca-49e2-4172-8d37-089b6c0295dd" (UID: "75611cca-49e2-4172-8d37-089b6c0295dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:30:45.434891 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.434862    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:45.498760 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.498725    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75611cca-49e2-4172-8d37-089b6c0295dd-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:45.583838 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.583807    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d30985d1-0ddd-486f-8104-94f3bdb4c049" path="/var/lib/kubelet/pods/d30985d1-0ddd-486f-8104-94f3bdb4c049/volumes"
Nov 02 12:30:45.863637 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.863609    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"]
Nov 02 12:30:45.863842 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:30:45.863817    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a7742a_f60b_4270_8992_2855b79bed42.slice/crio-93a02fd2e749581edda2cf619bc20d8f335479f1b3af458338115bbd86a4fbb7 WatchSource:0}: Error finding container 93a02fd2e749581edda2cf619bc20d8f335479f1b3af458338115bbd86a4fbb7: Status 404 returned error can't find the container with id 93a02fd2e749581edda2cf619bc20d8f335479f1b3af458338115bbd86a4fbb7
Nov 02 12:30:45.866320 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.866290    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:30:45.866388 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.866354    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:30:45.866388 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.866383    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:30:45.875891 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.875869    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"
Nov 02 12:30:45.875891 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.875878    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7" event={"ID":"75611cca-49e2-4172-8d37-089b6c0295dd","Type":"ContainerDied","Data":"e4d3594802ee2cde7f5c449f3e8e926dd6a5e4561f283b4000e39003306035c4"}
Nov 02 12:30:45.876074 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.875925    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4d3594802ee2cde7f5c449f3e8e926dd6a5e4561f283b4000e39003306035c4"
Nov 02 12:30:45.879672 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:45.879649    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j" event={"ID":"a7a7742a-f60b-4270-8992-2855b79bed42","Type":"ContainerStarted","Data":"93a02fd2e749581edda2cf619bc20d8f335479f1b3af458338115bbd86a4fbb7"}
Nov 02 12:30:46.887609 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:46.887573    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j" event={"ID":"a7a7742a-f60b-4270-8992-2855b79bed42","Type":"ContainerStarted","Data":"f81cb9696d14ff74d92bc0b637701470e44d76acca6b095b50336d6f61106064"}
Nov 02 12:30:47.064072 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:47.064014    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j" podStartSLOduration=3.064001517 podStartE2EDuration="3.064001517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:47.061235452 +0000 UTC m=+2075.624943862" watchObservedRunningTime="2026-11-02 12:30:47.064001517 +0000 UTC m=+2075.627709922"
Nov 02 12:30:47.435345 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:47.435315    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:47.440371 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:47.440343    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:47.895412 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:47.895380    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:47.896191 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:47.896170    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:30:48.071560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.071529    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"]
Nov 02 12:30:48.072266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.072245    4464 cpu_manager.go:414] "containerMap: removing container" podUID="75611cca-49e2-4172-8d37-089b6c0295dd" containerName="revoke-keys"
Nov 02 12:30:48.072266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.072265    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75611cca-49e2-4172-8d37-089b6c0295dd" containerName="revoke-keys"
Nov 02 12:30:48.072451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.072407    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="75611cca-49e2-4172-8d37-089b6c0295dd" containerName="revoke-keys"
Nov 02 12:30:48.076211 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.076187    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"
Nov 02 12:30:48.134617 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.133339    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:30:48.137577 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.137520    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:30:48.137577 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.137572    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:30:48.137818 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.137607    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:30:48.138581 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.138546    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503f6757-3f34-4003-9559-22244e0d3283-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-02304760-gt9cr\" (UID: \"503f6757-3f34-4003-9559-22244e0d3283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"
Nov 02 12:30:48.138908 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.138886    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrt47\" (UniqueName: \"kubernetes.io/projected/503f6757-3f34-4003-9559-22244e0d3283-kube-api-access-zrt47\") pod \"maas-api-revoke-keys-e2e-derive-02304760-gt9cr\" (UID: \"503f6757-3f34-4003-9559-22244e0d3283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"
Nov 02 12:30:48.139130 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.139113    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503f6757-3f34-4003-9559-22244e0d3283-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-02304760-gt9cr\" (UID: \"503f6757-3f34-4003-9559-22244e0d3283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"
Nov 02 12:30:48.141142 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.141118    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"]
Nov 02 12:30:48.240759 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.240732    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503f6757-3f34-4003-9559-22244e0d3283-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-02304760-gt9cr\" (UID: \"503f6757-3f34-4003-9559-22244e0d3283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"
Nov 02 12:30:48.240916 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.240901    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503f6757-3f34-4003-9559-22244e0d3283-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-02304760-gt9cr\" (UID: \"503f6757-3f34-4003-9559-22244e0d3283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"
Nov 02 12:30:48.241004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.240967    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrt47\" (UniqueName: \"kubernetes.io/projected/503f6757-3f34-4003-9559-22244e0d3283-kube-api-access-zrt47\") pod \"maas-api-revoke-keys-e2e-derive-02304760-gt9cr\" (UID: \"503f6757-3f34-4003-9559-22244e0d3283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"
Nov 02 12:30:48.241434 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.241412    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503f6757-3f34-4003-9559-22244e0d3283-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-02304760-gt9cr\" (UID: \"503f6757-3f34-4003-9559-22244e0d3283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"
Nov 02 12:30:48.243255 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.243233    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503f6757-3f34-4003-9559-22244e0d3283-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-02304760-gt9cr\" (UID: \"503f6757-3f34-4003-9559-22244e0d3283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"
Nov 02 12:30:48.260158 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.260133    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h" podUID="e85c3283-ce74-4e7f-b191-79e497d42a77" containerName="payload-processing" containerID="cri-o://323ed9066018c5b0096dc22e3b6963b146e3f7934e16502372f0798dac762216" gracePeriod=25
Nov 02 12:30:48.303112 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.303085    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrt47\" (UniqueName: \"kubernetes.io/projected/503f6757-3f34-4003-9559-22244e0d3283-kube-api-access-zrt47\") pod \"maas-api-revoke-keys-e2e-derive-02304760-gt9cr\" (UID: \"503f6757-3f34-4003-9559-22244e0d3283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"
Nov 02 12:30:48.310961 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.310934    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l" podUID="964e4b45-70c3-4ab2-8960-cf5f56c36a04" containerName="payload-pre-processing" containerID="cri-o://db6d4de928c1321b1f6297cb9d10d4162c5fdc6f62931b10d3b758864ca4d128" gracePeriod=25
Nov 02 12:30:48.400303 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.400272    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"
Nov 02 12:30:48.991235 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:48.991103    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"]
Nov 02 12:30:48.995137 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:30:48.995106    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503f6757_3f34_4003_9559_22244e0d3283.slice/crio-07162122bdd2c952b861f3d16f67b234272120a46e5596eda49e81a8a9cb70ad WatchSource:0}: Error finding container 07162122bdd2c952b861f3d16f67b234272120a46e5596eda49e81a8a9cb70ad: Status 404 returned error can't find the container with id 07162122bdd2c952b861f3d16f67b234272120a46e5596eda49e81a8a9cb70ad
Nov 02 12:30:49.944042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:49.943963    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr" event={"ID":"503f6757-3f34-4003-9559-22244e0d3283","Type":"ContainerStarted","Data":"f2380cb4c920f12cbf1ece26523f00eb759eac352b4ca42d320b74c8cd0216c2"}
Nov 02 12:30:49.944220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:49.944070    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr" event={"ID":"503f6757-3f34-4003-9559-22244e0d3283","Type":"ContainerStarted","Data":"07162122bdd2c952b861f3d16f67b234272120a46e5596eda49e81a8a9cb70ad"}
Nov 02 12:30:50.071090 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.071016    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr" podStartSLOduration=2.070997222 podStartE2EDuration="2.070997222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:50.068463959 +0000 UTC m=+2078.632172370" watchObservedRunningTime="2026-11-02 12:30:50.070997222 +0000 UTC m=+2078.634705632"
Nov 02 12:30:50.517074 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.517019    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"]
Nov 02 12:30:50.527046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.521947    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.609266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.608520    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-data\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.609266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.608697    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-token\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.609266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.608734    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-podinfo\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.609266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.608843    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-workload-certs\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.609266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.608876    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666a9bf6-5863-44ce-9574-26ca5ee9b856-istiod-ca-cert\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.609266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.608911    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-workload-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.609266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.608953    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8xzh\" (UniqueName: \"kubernetes.io/projected/666a9bf6-5863-44ce-9574-26ca5ee9b856-kube-api-access-g8xzh\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.609266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.609005    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-envoy\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.609266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.609037    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-credential-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.631062 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.631029    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"]
Nov 02 12:30:50.710211 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.710178    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666a9bf6-5863-44ce-9574-26ca5ee9b856-istiod-ca-cert\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.710382 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.710218    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-workload-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.710382 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.710254    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8xzh\" (UniqueName: \"kubernetes.io/projected/666a9bf6-5863-44ce-9574-26ca5ee9b856-kube-api-access-g8xzh\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.710382 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.710279    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-envoy\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.710382 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.710301    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-credential-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.710693 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.710456    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-data\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.710693 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.710598    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-token\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.710693 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.710627    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-podinfo\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.710865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.710710    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-credential-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.710865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.710733    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-workload-certs\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.710865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.710800    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-workload-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.711008 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.710862    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-data\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.711082 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.711062    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666a9bf6-5863-44ce-9574-26ca5ee9b856-istiod-ca-cert\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.711132 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.711117    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-workload-certs\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.712600 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.712572    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-envoy\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.712927 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.712910    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-podinfo\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.777127 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.777057    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8xzh\" (UniqueName: \"kubernetes.io/projected/666a9bf6-5863-44ce-9574-26ca5ee9b856-kube-api-access-g8xzh\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.777461 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.777443    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-token\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.840415 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.840380    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:50.953340 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.953306    4464 generic.go:356] "Generic (PLEG): container finished" podID="503f6757-3f34-4003-9559-22244e0d3283" containerID="f2380cb4c920f12cbf1ece26523f00eb759eac352b4ca42d320b74c8cd0216c2" exitCode=0
Nov 02 12:30:50.953544 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:50.953498    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr" event={"ID":"503f6757-3f34-4003-9559-22244e0d3283","Type":"ContainerDied","Data":"f2380cb4c920f12cbf1ece26523f00eb759eac352b4ca42d320b74c8cd0216c2"}
Nov 02 12:30:51.270316 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:30:51.270284    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod666a9bf6_5863_44ce_9574_26ca5ee9b856.slice/crio-1542f90e6705ad5954ea1f9a35f9295d064cfa097f9529b3aea8b2c61c428c4b WatchSource:0}: Error finding container 1542f90e6705ad5954ea1f9a35f9295d064cfa097f9529b3aea8b2c61c428c4b: Status 404 returned error can't find the container with id 1542f90e6705ad5954ea1f9a35f9295d064cfa097f9529b3aea8b2c61c428c4b
Nov 02 12:30:51.270764 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:51.270604    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"]
Nov 02 12:30:51.272874 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:51.272844    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:30:51.272947 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:51.272918    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:30:51.273054 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:51.272947    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:30:51.987000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:51.984413    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh" event={"ID":"666a9bf6-5863-44ce-9574-26ca5ee9b856","Type":"ContainerStarted","Data":"8b7b6d5ee62e12ba0477f445fff4b9510c399976a5a8863d39bd85444af4d551"}
Nov 02 12:30:51.987000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:51.984478    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh" event={"ID":"666a9bf6-5863-44ce-9574-26ca5ee9b856","Type":"ContainerStarted","Data":"1542f90e6705ad5954ea1f9a35f9295d064cfa097f9529b3aea8b2c61c428c4b"}
Nov 02 12:30:52.111161 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:52.111101    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh" podStartSLOduration=2.111085654 podStartE2EDuration="2.111085654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:52.107448806 +0000 UTC m=+2080.671157216" watchObservedRunningTime="2026-11-02 12:30:52.111085654 +0000 UTC m=+2080.674794066"
Nov 02 12:30:52.157899 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:52.157872    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"
Nov 02 12:30:52.233241 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:52.233208    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503f6757-3f34-4003-9559-22244e0d3283-kube-api-access-zrt47\" (UniqueName: \"kubernetes.io/projected/503f6757-3f34-4003-9559-22244e0d3283-kube-api-access-zrt47\") pod \"503f6757-3f34-4003-9559-22244e0d3283\" (UID: \"503f6757-3f34-4003-9559-22244e0d3283\") "
Nov 02 12:30:52.233448 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:52.233266    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/503f6757-3f34-4003-9559-22244e0d3283-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503f6757-3f34-4003-9559-22244e0d3283-maas-api-openshift-ca\") pod \"503f6757-3f34-4003-9559-22244e0d3283\" (UID: \"503f6757-3f34-4003-9559-22244e0d3283\") "
Nov 02 12:30:52.233448 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:52.233329    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/503f6757-3f34-4003-9559-22244e0d3283-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503f6757-3f34-4003-9559-22244e0d3283-maas-api-xks-ca\") pod \"503f6757-3f34-4003-9559-22244e0d3283\" (UID: \"503f6757-3f34-4003-9559-22244e0d3283\") "
Nov 02 12:30:52.233696 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:52.233671    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/503f6757-3f34-4003-9559-22244e0d3283-maas-api-openshift-ca" pod "503f6757-3f34-4003-9559-22244e0d3283" (UID: "503f6757-3f34-4003-9559-22244e0d3283"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:30:52.234241 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:52.234216    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503f6757-3f34-4003-9559-22244e0d3283-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:52.235436 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:52.235398    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/503f6757-3f34-4003-9559-22244e0d3283-maas-api-xks-ca" pod "503f6757-3f34-4003-9559-22244e0d3283" (UID: "503f6757-3f34-4003-9559-22244e0d3283"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:30:52.235536 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:52.235474    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503f6757-3f34-4003-9559-22244e0d3283-kube-api-access-zrt47" pod "503f6757-3f34-4003-9559-22244e0d3283" (UID: "503f6757-3f34-4003-9559-22244e0d3283"). InnerVolumeSpecName "kube-api-access-zrt47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:52.335314 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:52.335281    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrt47\" (UniqueName: \"kubernetes.io/projected/503f6757-3f34-4003-9559-22244e0d3283-kube-api-access-zrt47\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:52.335314 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:52.335310    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503f6757-3f34-4003-9559-22244e0d3283-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:52.840922 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:52.840890    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:52.842432 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:52.842403    4464 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 02 12:30:52.842610 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:52.842478    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh" podUID="666a9bf6-5863-44ce-9574-26ca5ee9b856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 02 12:30:53.015002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:53.012107    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"
Nov 02 12:30:53.015002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:53.012915    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr" event={"ID":"503f6757-3f34-4003-9559-22244e0d3283","Type":"ContainerDied","Data":"07162122bdd2c952b861f3d16f67b234272120a46e5596eda49e81a8a9cb70ad"}
Nov 02 12:30:53.015002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:53.013069    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07162122bdd2c952b861f3d16f67b234272120a46e5596eda49e81a8a9cb70ad"
Nov 02 12:30:53.841141 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:53.841102    4464 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 02 12:30:53.841538 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:53.841166    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh" podUID="666a9bf6-5863-44ce-9574-26ca5ee9b856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 02 12:30:54.841405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:54.841371    4464 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 02 12:30:54.841785 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:54.841424    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh" podUID="666a9bf6-5863-44ce-9574-26ca5ee9b856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 02 12:30:55.844442 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:55.844405    4464 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 02 12:30:55.845064 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:55.845030    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh" podUID="666a9bf6-5863-44ce-9574-26ca5ee9b856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 02 12:30:56.425202 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.419228    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"]
Nov 02 12:30:56.425202 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.422884    4464 cpu_manager.go:414] "containerMap: removing container" podUID="503f6757-3f34-4003-9559-22244e0d3283" containerName="revoke-keys"
Nov 02 12:30:56.425202 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.422909    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503f6757-3f34-4003-9559-22244e0d3283" containerName="revoke-keys"
Nov 02 12:30:56.425202 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.423057    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="503f6757-3f34-4003-9559-22244e0d3283" containerName="revoke-keys"
Nov 02 12:30:56.427389 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.427360    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"]
Nov 02 12:30:56.427580 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.427366    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"
Nov 02 12:30:56.534956 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.534924    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-cf0095-dockercfg-wgk7b\""
Nov 02 12:30:56.535149 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.534960    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-cf0095\""
Nov 02 12:30:56.548192 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.548164    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htrsz\" (UniqueName: \"kubernetes.io/projected/a3d1d64c-7010-410b-ac86-54b9d481e8ac-kube-api-access-htrsz\") pod \"payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv\" (UID: \"a3d1d64c-7010-410b-ac86-54b9d481e8ac\") " pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"
Nov 02 12:30:56.548343 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.548302    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d1d64c-7010-410b-ac86-54b9d481e8ac-plugins-config-volume\") pod \"payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv\" (UID: \"a3d1d64c-7010-410b-ac86-54b9d481e8ac\") " pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"
Nov 02 12:30:56.649748 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.649711    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d1d64c-7010-410b-ac86-54b9d481e8ac-plugins-config-volume\") pod \"payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv\" (UID: \"a3d1d64c-7010-410b-ac86-54b9d481e8ac\") " pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"
Nov 02 12:30:56.649930 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.649822    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htrsz\" (UniqueName: \"kubernetes.io/projected/a3d1d64c-7010-410b-ac86-54b9d481e8ac-kube-api-access-htrsz\") pod \"payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv\" (UID: \"a3d1d64c-7010-410b-ac86-54b9d481e8ac\") " pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"
Nov 02 12:30:56.650422 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.650398    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d1d64c-7010-410b-ac86-54b9d481e8ac-plugins-config-volume\") pod \"payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv\" (UID: \"a3d1d64c-7010-410b-ac86-54b9d481e8ac\") " pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"
Nov 02 12:30:56.712476 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.712403    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htrsz\" (UniqueName: \"kubernetes.io/projected/a3d1d64c-7010-410b-ac86-54b9d481e8ac-kube-api-access-htrsz\") pod \"payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv\" (UID: \"a3d1d64c-7010-410b-ac86-54b9d481e8ac\") " pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"
Nov 02 12:30:56.742303 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.742266    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"
Nov 02 12:30:56.798413 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.798383    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l" podUID="964e4b45-70c3-4ab2-8960-cf5f56c36a04" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:30:56.798846 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.798812    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h" podUID="e85c3283-ce74-4e7f-b191-79e497d42a77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:30:56.845398 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.845364    4464 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 02 12:30:56.845808 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:56.845436    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh" podUID="666a9bf6-5863-44ce-9574-26ca5ee9b856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 02 12:30:57.147648 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:57.147617    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"]
Nov 02 12:30:57.147823 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:30:57.147799    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d1d64c_7010_410b_ac86_54b9d481e8ac.slice/crio-28bcf2085278e327e3d337b165c4baacb648370da013b7302f936510ee576987 WatchSource:0}: Error finding container 28bcf2085278e327e3d337b165c4baacb648370da013b7302f936510ee576987: Status 404 returned error can't find the container with id 28bcf2085278e327e3d337b165c4baacb648370da013b7302f936510ee576987
Nov 02 12:30:57.895144 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:57.895115    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:57.895607 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:57.895577    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:57.896633 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:57.896612    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:30:58.098320 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:58.098282    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv" event={"ID":"a3d1d64c-7010-410b-ac86-54b9d481e8ac","Type":"ContainerStarted","Data":"3ad150c0ec7297fc455491e17d1193140b702cdd20d9265702d97a8085cbea76"}
Nov 02 12:30:58.098477 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:58.098344    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv" event={"ID":"a3d1d64c-7010-410b-ac86-54b9d481e8ac","Type":"ContainerStarted","Data":"28bcf2085278e327e3d337b165c4baacb648370da013b7302f936510ee576987"}
Nov 02 12:30:58.098477 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:58.098374    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"
Nov 02 12:30:58.261149 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:58.261100    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv" podStartSLOduration=2.26108585 podStartE2EDuration="2.26108585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:58.259527124 +0000 UTC m=+2086.823235534" watchObservedRunningTime="2026-11-02 12:30:58.26108585 +0000 UTC m=+2086.824794278"
Nov 02 12:30:58.292310 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:58.292277    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:30:58.733211 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:58.733171    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm" podUID="8eb017b2-26e3-450a-953f-e348a1109d20" containerName="main" containerID="cri-o://1f1c2f806321de151e09bff07748ece1107cb55d033d54769ea19a6c3f0e2a12" gracePeriod=45
Nov 02 12:30:59.008925 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.008898    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:30:59.106628 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.106600    4464 generic.go:356] "Generic (PLEG): container finished" podID="8eb017b2-26e3-450a-953f-e348a1109d20" containerID="1f1c2f806321de151e09bff07748ece1107cb55d033d54769ea19a6c3f0e2a12" exitCode=0
Nov 02 12:30:59.106790 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.106650    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"
Nov 02 12:30:59.106790 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.106687    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm" event={"ID":"8eb017b2-26e3-450a-953f-e348a1109d20","Type":"ContainerDied","Data":"1f1c2f806321de151e09bff07748ece1107cb55d033d54769ea19a6c3f0e2a12"}
Nov 02 12:30:59.106790 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.106730    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm" event={"ID":"8eb017b2-26e3-450a-953f-e348a1109d20","Type":"ContainerDied","Data":"6cee3b0e4e47896e92e35b1d7fba0ddfaebeef1edcde2d07de03b270c8971eea"}
Nov 02 12:30:59.106790 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.106753    4464 scope.go:122] "RemoveContainer" containerID="1f1c2f806321de151e09bff07748ece1107cb55d033d54769ea19a6c3f0e2a12"
Nov 02 12:30:59.114118 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.114099    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb017b2-26e3-450a-953f-e348a1109d20-kube-api-access-84mbx\" (UniqueName: \"kubernetes.io/projected/8eb017b2-26e3-450a-953f-e348a1109d20-kube-api-access-84mbx\") pod \"8eb017b2-26e3-450a-953f-e348a1109d20\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") "
Nov 02 12:30:59.114217 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.114142    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-model-cache\") pod \"8eb017b2-26e3-450a-953f-e348a1109d20\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") "
Nov 02 12:30:59.114266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.114216    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eb017b2-26e3-450a-953f-e348a1109d20-tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb017b2-26e3-450a-953f-e348a1109d20-tls-certs\") pod \"8eb017b2-26e3-450a-953f-e348a1109d20\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") "
Nov 02 12:30:59.114266 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.114261    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-home\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-home\") pod \"8eb017b2-26e3-450a-953f-e348a1109d20\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") "
Nov 02 12:30:59.114415 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.114295    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-dshm\") pod \"8eb017b2-26e3-450a-953f-e348a1109d20\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") "
Nov 02 12:30:59.114505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.114474    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-model-cache" pod "8eb017b2-26e3-450a-953f-e348a1109d20" (UID: "8eb017b2-26e3-450a-953f-e348a1109d20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:59.114602 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.114580    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-tmp-dir\") pod \"8eb017b2-26e3-450a-953f-e348a1109d20\" (UID: \"8eb017b2-26e3-450a-953f-e348a1109d20\") "
Nov 02 12:30:59.114822 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.114799    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-tmp-dir" pod "8eb017b2-26e3-450a-953f-e348a1109d20" (UID: "8eb017b2-26e3-450a-953f-e348a1109d20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:59.114920 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.114902    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-home" pod "8eb017b2-26e3-450a-953f-e348a1109d20" (UID: "8eb017b2-26e3-450a-953f-e348a1109d20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:59.115036 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.115019    4464 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-tmp-dir\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:59.115110 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.115039    4464 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-model-cache\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:59.115110 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.115051    4464 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-home\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:59.116517 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.116411    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-dshm" pod "8eb017b2-26e3-450a-953f-e348a1109d20" (UID: "8eb017b2-26e3-450a-953f-e348a1109d20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:30:59.116597 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.116546    4464 scope.go:122] "RemoveContainer" containerID="1f1c2f806321de151e09bff07748ece1107cb55d033d54769ea19a6c3f0e2a12"
Nov 02 12:30:59.116597 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.116581    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eb017b2-26e3-450a-953f-e348a1109d20-tls-certs" pod "8eb017b2-26e3-450a-953f-e348a1109d20" (UID: "8eb017b2-26e3-450a-953f-e348a1109d20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:30:59.116687 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.116601    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb017b2-26e3-450a-953f-e348a1109d20-kube-api-access-84mbx" pod "8eb017b2-26e3-450a-953f-e348a1109d20" (UID: "8eb017b2-26e3-450a-953f-e348a1109d20"). InnerVolumeSpecName "kube-api-access-84mbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:30:59.116850 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:30:59.116826    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f1c2f806321de151e09bff07748ece1107cb55d033d54769ea19a6c3f0e2a12\": container with ID starting with 1f1c2f806321de151e09bff07748ece1107cb55d033d54769ea19a6c3f0e2a12 not found: ID does not exist" containerID="1f1c2f806321de151e09bff07748ece1107cb55d033d54769ea19a6c3f0e2a12"
Nov 02 12:30:59.116891 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.116861    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f1c2f806321de151e09bff07748ece1107cb55d033d54769ea19a6c3f0e2a12"} err="failed to get container status \"1f1c2f806321de151e09bff07748ece1107cb55d033d54769ea19a6c3f0e2a12\": rpc error: code = NotFound desc = could not find container \"1f1c2f806321de151e09bff07748ece1107cb55d033d54769ea19a6c3f0e2a12\": container with ID starting with 1f1c2f806321de151e09bff07748ece1107cb55d033d54769ea19a6c3f0e2a12 not found: ID does not exist"
Nov 02 12:30:59.216845 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.216812    4464 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb017b2-26e3-450a-953f-e348a1109d20-tls-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:59.216845 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.216841    4464 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb017b2-26e3-450a-953f-e348a1109d20-dshm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:59.217074 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.216858    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84mbx\" (UniqueName: \"kubernetes.io/projected/8eb017b2-26e3-450a-953f-e348a1109d20-kube-api-access-84mbx\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:30:59.378933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.378852    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"]
Nov 02 12:30:59.485560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.485528    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"]
Nov 02 12:30:59.730566 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.730524    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"]
Nov 02 12:30:59.733816 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.733792    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm"]
Nov 02 12:30:59.971244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:30:59.969074    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-03c4e8/sub-test-model-03c4e8-kserve-7fb87cdc59-v59pm" podUID="8eb017b2-26e3-450a-953f-e348a1109d20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": context deadline exceeded"
Nov 02 12:31:01.582951 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:01.582911    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb017b2-26e3-450a-953f-e348a1109d20" path="/var/lib/kubelet/pods/8eb017b2-26e3-450a-953f-e348a1109d20/volumes"
Nov 02 12:31:04.386847 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:04.386370    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r" podUID="c1e079e8-6acf-41b9-b8a8-1a066acccb93" containerName="payload-processing" containerID="cri-o://22457be605d51c4e0997d215de5f6347b140051197777dfabfd16241ef87ce6f" gracePeriod=25
Nov 02 12:31:04.486253 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:04.486188    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t" podUID="d89ec611-1f60-49e5-80b9-ef7720ecb732" containerName="payload-pre-processing" containerID="cri-o://72d49a39903200e08304e614d95f74127f546775c905bab0f0bf49708a30e0a4" gracePeriod=25
Nov 02 12:31:04.743718 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:04.743680    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"
Nov 02 12:31:04.748348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:04.747707    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1e079e8-6acf-41b9-b8a8-1a066acccb93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e079e8-6acf-41b9-b8a8-1a066acccb93-plugins-config-volume\") pod \"c1e079e8-6acf-41b9-b8a8-1a066acccb93\" (UID: \"c1e079e8-6acf-41b9-b8a8-1a066acccb93\") "
Nov 02 12:31:04.748348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:04.747790    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e079e8-6acf-41b9-b8a8-1a066acccb93-kube-api-access-6qch9\" (UniqueName: \"kubernetes.io/projected/c1e079e8-6acf-41b9-b8a8-1a066acccb93-kube-api-access-6qch9\") pod \"c1e079e8-6acf-41b9-b8a8-1a066acccb93\" (UID: \"c1e079e8-6acf-41b9-b8a8-1a066acccb93\") "
Nov 02 12:31:04.748348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:04.748172    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e079e8-6acf-41b9-b8a8-1a066acccb93-plugins-config-volume" pod "c1e079e8-6acf-41b9-b8a8-1a066acccb93" (UID: "c1e079e8-6acf-41b9-b8a8-1a066acccb93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:31:04.748348 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:04.748318    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1e079e8-6acf-41b9-b8a8-1a066acccb93-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:04.750197 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:04.750171    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e079e8-6acf-41b9-b8a8-1a066acccb93-kube-api-access-6qch9" pod "c1e079e8-6acf-41b9-b8a8-1a066acccb93" (UID: "c1e079e8-6acf-41b9-b8a8-1a066acccb93"). InnerVolumeSpecName "kube-api-access-6qch9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:31:04.773118 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:04.773098    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"
Nov 02 12:31:04.849414 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:04.849384    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89ec611-1f60-49e5-80b9-ef7720ecb732-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89ec611-1f60-49e5-80b9-ef7720ecb732-plugins-config-volume\") pod \"d89ec611-1f60-49e5-80b9-ef7720ecb732\" (UID: \"d89ec611-1f60-49e5-80b9-ef7720ecb732\") "
Nov 02 12:31:04.849549 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:04.849460    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89ec611-1f60-49e5-80b9-ef7720ecb732-kube-api-access-289gg\" (UniqueName: \"kubernetes.io/projected/d89ec611-1f60-49e5-80b9-ef7720ecb732-kube-api-access-289gg\") pod \"d89ec611-1f60-49e5-80b9-ef7720ecb732\" (UID: \"d89ec611-1f60-49e5-80b9-ef7720ecb732\") "
Nov 02 12:31:04.849732 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:04.849709    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89ec611-1f60-49e5-80b9-ef7720ecb732-plugins-config-volume" pod "d89ec611-1f60-49e5-80b9-ef7720ecb732" (UID: "d89ec611-1f60-49e5-80b9-ef7720ecb732"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:31:04.849842 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:04.849830    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d89ec611-1f60-49e5-80b9-ef7720ecb732-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:04.849884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:04.849845    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qch9\" (UniqueName: \"kubernetes.io/projected/c1e079e8-6acf-41b9-b8a8-1a066acccb93-kube-api-access-6qch9\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:04.851445 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:04.851427    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89ec611-1f60-49e5-80b9-ef7720ecb732-kube-api-access-289gg" pod "d89ec611-1f60-49e5-80b9-ef7720ecb732" (UID: "d89ec611-1f60-49e5-80b9-ef7720ecb732"). InnerVolumeSpecName "kube-api-access-289gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:31:04.950940 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:04.950859    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-289gg\" (UniqueName: \"kubernetes.io/projected/d89ec611-1f60-49e5-80b9-ef7720ecb732-kube-api-access-289gg\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:05.152095 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.152060    4464 generic.go:356] "Generic (PLEG): container finished" podID="d89ec611-1f60-49e5-80b9-ef7720ecb732" containerID="72d49a39903200e08304e614d95f74127f546775c905bab0f0bf49708a30e0a4" exitCode=0
Nov 02 12:31:05.152239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.152104    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"
Nov 02 12:31:05.152239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.152141    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t" event={"ID":"d89ec611-1f60-49e5-80b9-ef7720ecb732","Type":"ContainerDied","Data":"72d49a39903200e08304e614d95f74127f546775c905bab0f0bf49708a30e0a4"}
Nov 02 12:31:05.152239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.152197    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t" event={"ID":"d89ec611-1f60-49e5-80b9-ef7720ecb732","Type":"ContainerDied","Data":"8eba6f77dd460ac57d696d8e36d6a9c20470fb38740774f4e16175114ac4f816"}
Nov 02 12:31:05.152239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.152229    4464 scope.go:122] "RemoveContainer" containerID="72d49a39903200e08304e614d95f74127f546775c905bab0f0bf49708a30e0a4"
Nov 02 12:31:05.153816 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.153781    4464 generic.go:356] "Generic (PLEG): container finished" podID="c1e079e8-6acf-41b9-b8a8-1a066acccb93" containerID="22457be605d51c4e0997d215de5f6347b140051197777dfabfd16241ef87ce6f" exitCode=0
Nov 02 12:31:05.153954 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.153818    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"
Nov 02 12:31:05.153954 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.153833    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r" event={"ID":"c1e079e8-6acf-41b9-b8a8-1a066acccb93","Type":"ContainerDied","Data":"22457be605d51c4e0997d215de5f6347b140051197777dfabfd16241ef87ce6f"}
Nov 02 12:31:05.153954 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.153864    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r" event={"ID":"c1e079e8-6acf-41b9-b8a8-1a066acccb93","Type":"ContainerDied","Data":"d09f1ae389f16a8334267c06c3733e0859b2b45810cc9d770b84300690b79f5b"}
Nov 02 12:31:05.164560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.164541    4464 scope.go:122] "RemoveContainer" containerID="72d49a39903200e08304e614d95f74127f546775c905bab0f0bf49708a30e0a4"
Nov 02 12:31:05.164831 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:31:05.164801    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d49a39903200e08304e614d95f74127f546775c905bab0f0bf49708a30e0a4\": container with ID starting with 72d49a39903200e08304e614d95f74127f546775c905bab0f0bf49708a30e0a4 not found: ID does not exist" containerID="72d49a39903200e08304e614d95f74127f546775c905bab0f0bf49708a30e0a4"
Nov 02 12:31:05.164941 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.164839    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d49a39903200e08304e614d95f74127f546775c905bab0f0bf49708a30e0a4"} err="failed to get container status \"72d49a39903200e08304e614d95f74127f546775c905bab0f0bf49708a30e0a4\": rpc error: code = NotFound desc = could not find container \"72d49a39903200e08304e614d95f74127f546775c905bab0f0bf49708a30e0a4\": container with ID starting with 72d49a39903200e08304e614d95f74127f546775c905bab0f0bf49708a30e0a4 not found: ID does not exist"
Nov 02 12:31:05.164941 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.164864    4464 scope.go:122] "RemoveContainer" containerID="22457be605d51c4e0997d215de5f6347b140051197777dfabfd16241ef87ce6f"
Nov 02 12:31:05.173856 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.173841    4464 scope.go:122] "RemoveContainer" containerID="22457be605d51c4e0997d215de5f6347b140051197777dfabfd16241ef87ce6f"
Nov 02 12:31:05.174116 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:31:05.174101    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22457be605d51c4e0997d215de5f6347b140051197777dfabfd16241ef87ce6f\": container with ID starting with 22457be605d51c4e0997d215de5f6347b140051197777dfabfd16241ef87ce6f not found: ID does not exist" containerID="22457be605d51c4e0997d215de5f6347b140051197777dfabfd16241ef87ce6f"
Nov 02 12:31:05.174167 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.174122    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22457be605d51c4e0997d215de5f6347b140051197777dfabfd16241ef87ce6f"} err="failed to get container status \"22457be605d51c4e0997d215de5f6347b140051197777dfabfd16241ef87ce6f\": rpc error: code = NotFound desc = could not find container \"22457be605d51c4e0997d215de5f6347b140051197777dfabfd16241ef87ce6f\": container with ID starting with 22457be605d51c4e0997d215de5f6347b140051197777dfabfd16241ef87ce6f not found: ID does not exist"
Nov 02 12:31:05.320596 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.320563    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"]
Nov 02 12:31:05.324104 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.324079    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-02304760-6f86dfbb8c-f554r"]
Nov 02 12:31:05.435337 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.435304    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"]
Nov 02 12:31:05.441409 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.441386    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-02304760-86c4f9d7f-v786t"]
Nov 02 12:31:05.582666 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.582579    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1e079e8-6acf-41b9-b8a8-1a066acccb93" path="/var/lib/kubelet/pods/c1e079e8-6acf-41b9-b8a8-1a066acccb93/volumes"
Nov 02 12:31:05.583102 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:05.583082    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89ec611-1f60-49e5-80b9-ef7720ecb732" path="/var/lib/kubelet/pods/d89ec611-1f60-49e5-80b9-ef7720ecb732/volumes"
Nov 02 12:31:06.800703 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:06.800667    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l" podUID="964e4b45-70c3-4ab2-8960-cf5f56c36a04" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:31:06.801248 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:06.800893    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h" podUID="e85c3283-ce74-4e7f-b191-79e497d42a77" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:31:08.576615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:08.576585    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"
Nov 02 12:31:08.602939 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:08.602912    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e85c3283-ce74-4e7f-b191-79e497d42a77-kube-api-access-7pn54\" (UniqueName: \"kubernetes.io/projected/e85c3283-ce74-4e7f-b191-79e497d42a77-kube-api-access-7pn54\") pod \"e85c3283-ce74-4e7f-b191-79e497d42a77\" (UID: \"e85c3283-ce74-4e7f-b191-79e497d42a77\") "
Nov 02 12:31:08.603129 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:08.602989    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e85c3283-ce74-4e7f-b191-79e497d42a77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e85c3283-ce74-4e7f-b191-79e497d42a77-plugins-config-volume\") pod \"e85c3283-ce74-4e7f-b191-79e497d42a77\" (UID: \"e85c3283-ce74-4e7f-b191-79e497d42a77\") "
Nov 02 12:31:08.603727 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:08.603428    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85c3283-ce74-4e7f-b191-79e497d42a77-plugins-config-volume" pod "e85c3283-ce74-4e7f-b191-79e497d42a77" (UID: "e85c3283-ce74-4e7f-b191-79e497d42a77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:31:08.605299 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:08.605276    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85c3283-ce74-4e7f-b191-79e497d42a77-kube-api-access-7pn54" pod "e85c3283-ce74-4e7f-b191-79e497d42a77" (UID: "e85c3283-ce74-4e7f-b191-79e497d42a77"). InnerVolumeSpecName "kube-api-access-7pn54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:31:08.622269 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:08.622251    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"
Nov 02 12:31:08.704885 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:08.704857    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/964e4b45-70c3-4ab2-8960-cf5f56c36a04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964e4b45-70c3-4ab2-8960-cf5f56c36a04-plugins-config-volume\") pod \"964e4b45-70c3-4ab2-8960-cf5f56c36a04\" (UID: \"964e4b45-70c3-4ab2-8960-cf5f56c36a04\") "
Nov 02 12:31:08.705106 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:08.704956    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/964e4b45-70c3-4ab2-8960-cf5f56c36a04-kube-api-access-8zff8\" (UniqueName: \"kubernetes.io/projected/964e4b45-70c3-4ab2-8960-cf5f56c36a04-kube-api-access-8zff8\") pod \"964e4b45-70c3-4ab2-8960-cf5f56c36a04\" (UID: \"964e4b45-70c3-4ab2-8960-cf5f56c36a04\") "
Nov 02 12:31:08.705329 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:08.705304    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964e4b45-70c3-4ab2-8960-cf5f56c36a04-plugins-config-volume" pod "964e4b45-70c3-4ab2-8960-cf5f56c36a04" (UID: "964e4b45-70c3-4ab2-8960-cf5f56c36a04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:31:08.705329 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:08.705320    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pn54\" (UniqueName: \"kubernetes.io/projected/e85c3283-ce74-4e7f-b191-79e497d42a77-kube-api-access-7pn54\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:08.705461 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:08.705345    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e85c3283-ce74-4e7f-b191-79e497d42a77-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:08.707064 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:08.707036    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964e4b45-70c3-4ab2-8960-cf5f56c36a04-kube-api-access-8zff8" pod "964e4b45-70c3-4ab2-8960-cf5f56c36a04" (UID: "964e4b45-70c3-4ab2-8960-cf5f56c36a04"). InnerVolumeSpecName "kube-api-access-8zff8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:31:08.806827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:08.806760    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zff8\" (UniqueName: \"kubernetes.io/projected/964e4b45-70c3-4ab2-8960-cf5f56c36a04-kube-api-access-8zff8\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:08.806827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:08.806785    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964e4b45-70c3-4ab2-8960-cf5f56c36a04-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:09.111066 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.110990    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"
Nov 02 12:31:09.210202 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.210152    4464 generic.go:356] "Generic (PLEG): container finished" podID="964e4b45-70c3-4ab2-8960-cf5f56c36a04" containerID="db6d4de928c1321b1f6297cb9d10d4162c5fdc6f62931b10d3b758864ca4d128" exitCode=0
Nov 02 12:31:09.210392 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.210289    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l" event={"ID":"964e4b45-70c3-4ab2-8960-cf5f56c36a04","Type":"ContainerDied","Data":"db6d4de928c1321b1f6297cb9d10d4162c5fdc6f62931b10d3b758864ca4d128"}
Nov 02 12:31:09.210392 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.210342    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l" event={"ID":"964e4b45-70c3-4ab2-8960-cf5f56c36a04","Type":"ContainerDied","Data":"cff3501cecbf7d5ae55021efed8e05c5b188103b756c72377aaa7365732e5931"}
Nov 02 12:31:09.210392 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.210371    4464 scope.go:122] "RemoveContainer" containerID="db6d4de928c1321b1f6297cb9d10d4162c5fdc6f62931b10d3b758864ca4d128"
Nov 02 12:31:09.210574 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.210496    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"
Nov 02 12:31:09.214204 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.214169    4464 generic.go:356] "Generic (PLEG): container finished" podID="e85c3283-ce74-4e7f-b191-79e497d42a77" containerID="323ed9066018c5b0096dc22e3b6963b146e3f7934e16502372f0798dac762216" exitCode=0
Nov 02 12:31:09.214356 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.214234    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"
Nov 02 12:31:09.214356 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.214262    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h" event={"ID":"e85c3283-ce74-4e7f-b191-79e497d42a77","Type":"ContainerDied","Data":"323ed9066018c5b0096dc22e3b6963b146e3f7934e16502372f0798dac762216"}
Nov 02 12:31:09.214356 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.214323    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h" event={"ID":"e85c3283-ce74-4e7f-b191-79e497d42a77","Type":"ContainerDied","Data":"191060eb0c99d4bdde77fa2ef5a5b3df0a2c50b8e4d57ee6c3a7da232b2e8063"}
Nov 02 12:31:09.223740 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.223720    4464 scope.go:122] "RemoveContainer" containerID="db6d4de928c1321b1f6297cb9d10d4162c5fdc6f62931b10d3b758864ca4d128"
Nov 02 12:31:09.224159 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:31:09.224139    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db6d4de928c1321b1f6297cb9d10d4162c5fdc6f62931b10d3b758864ca4d128\": container with ID starting with db6d4de928c1321b1f6297cb9d10d4162c5fdc6f62931b10d3b758864ca4d128 not found: ID does not exist" containerID="db6d4de928c1321b1f6297cb9d10d4162c5fdc6f62931b10d3b758864ca4d128"
Nov 02 12:31:09.224218 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.224166    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db6d4de928c1321b1f6297cb9d10d4162c5fdc6f62931b10d3b758864ca4d128"} err="failed to get container status \"db6d4de928c1321b1f6297cb9d10d4162c5fdc6f62931b10d3b758864ca4d128\": rpc error: code = NotFound desc = could not find container \"db6d4de928c1321b1f6297cb9d10d4162c5fdc6f62931b10d3b758864ca4d128\": container with ID starting with db6d4de928c1321b1f6297cb9d10d4162c5fdc6f62931b10d3b758864ca4d128 not found: ID does not exist"
Nov 02 12:31:09.224218 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.224182    4464 scope.go:122] "RemoveContainer" containerID="323ed9066018c5b0096dc22e3b6963b146e3f7934e16502372f0798dac762216"
Nov 02 12:31:09.233718 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.233697    4464 scope.go:122] "RemoveContainer" containerID="323ed9066018c5b0096dc22e3b6963b146e3f7934e16502372f0798dac762216"
Nov 02 12:31:09.233995 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:31:09.233958    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323ed9066018c5b0096dc22e3b6963b146e3f7934e16502372f0798dac762216\": container with ID starting with 323ed9066018c5b0096dc22e3b6963b146e3f7934e16502372f0798dac762216 not found: ID does not exist" containerID="323ed9066018c5b0096dc22e3b6963b146e3f7934e16502372f0798dac762216"
Nov 02 12:31:09.234082 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.234003    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323ed9066018c5b0096dc22e3b6963b146e3f7934e16502372f0798dac762216"} err="failed to get container status \"323ed9066018c5b0096dc22e3b6963b146e3f7934e16502372f0798dac762216\": rpc error: code = NotFound desc = could not find container \"323ed9066018c5b0096dc22e3b6963b146e3f7934e16502372f0798dac762216\": container with ID starting with 323ed9066018c5b0096dc22e3b6963b146e3f7934e16502372f0798dac762216 not found: ID does not exist"
Nov 02 12:31:09.344676 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.344646    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"]
Nov 02 12:31:09.350577 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.350542    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1703c-5cb549d5b7-8f84h"]
Nov 02 12:31:09.480772 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.480741    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"]
Nov 02 12:31:09.487562 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.487527    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1703c-5d6cc6899f-99f2l"]
Nov 02 12:31:09.602105 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.594392    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="964e4b45-70c3-4ab2-8960-cf5f56c36a04" path="/var/lib/kubelet/pods/964e4b45-70c3-4ab2-8960-cf5f56c36a04/volumes"
Nov 02 12:31:09.602105 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.595197    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e85c3283-ce74-4e7f-b191-79e497d42a77" path="/var/lib/kubelet/pods/e85c3283-ce74-4e7f-b191-79e497d42a77/volumes"
Nov 02 12:31:09.614856 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.614659    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"]
Nov 02 12:31:09.615349 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.615318    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9" podUID="cad06365-a977-4f66-94eb-2511cc1d02f0" containerName="istio-proxy" containerID="cri-o://0922e9cf37b9b2fff62d9a673d3e8d456fb5de6c330e8826209f3bd53fc4c443" gracePeriod=30
Nov 02 12:31:09.622322 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.622297    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"]
Nov 02 12:31:09.622509 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:09.622488    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p" podUID="dda61e72-8ab1-4390-8295-83a8fc113578" containerName="istio-proxy" containerID="cri-o://d5aa3a11acdd3c1d9a91f07eb8c1c2f3d2ea4c1a9fe5333a29b2eaac8d901ead" gracePeriod=30
Nov 02 12:31:11.085047 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:11.085011    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:31:11.875588 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:11.875562    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:31:11.892933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:11.892905    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:31:11.906083 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:11.906057    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:31:11.928456 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:11.928434    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:31:13.357913 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:13.357878    4464 scope.go:122] "RemoveContainer" containerID="52a661decff4615686f8a4c9fec8b55bbf08932afdec776da427d94696bac20f"
Nov 02 12:31:14.989702 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:14.989657    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:31:15.014967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.014941    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:31:15.083382 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.083352    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-data\") pod \"cad06365-a977-4f66-94eb-2511cc1d02f0\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") "
Nov 02 12:31:15.083382 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.083387    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-token\" (UniqueName: \"kubernetes.io/projected/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-token\") pod \"cad06365-a977-4f66-94eb-2511cc1d02f0\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") "
Nov 02 12:31:15.083645 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.083422    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad06365-a977-4f66-94eb-2511cc1d02f0-kube-api-access-2q5l2\" (UniqueName: \"kubernetes.io/projected/cad06365-a977-4f66-94eb-2511cc1d02f0-kube-api-access-2q5l2\") pod \"cad06365-a977-4f66-94eb-2511cc1d02f0\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") "
Nov 02 12:31:15.083645 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.083449    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-podinfo\") pod \"cad06365-a977-4f66-94eb-2511cc1d02f0\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") "
Nov 02 12:31:15.083645 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.083484    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-workload-certs\") pod \"dda61e72-8ab1-4390-8295-83a8fc113578\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") "
Nov 02 12:31:15.083645 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.083530    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-credential-socket\") pod \"cad06365-a977-4f66-94eb-2511cc1d02f0\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") "
Nov 02 12:31:15.083645 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.083565    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-istio-envoy\") pod \"dda61e72-8ab1-4390-8295-83a8fc113578\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") "
Nov 02 12:31:15.083645 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.083621    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad06365-a977-4f66-94eb-2511cc1d02f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad06365-a977-4f66-94eb-2511cc1d02f0-istiod-ca-cert\") pod \"cad06365-a977-4f66-94eb-2511cc1d02f0\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") "
Nov 02 12:31:15.084005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.083671    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-workload-socket\") pod \"dda61e72-8ab1-4390-8295-83a8fc113578\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") "
Nov 02 12:31:15.084005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.083717    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-workload-socket\") pod \"cad06365-a977-4f66-94eb-2511cc1d02f0\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") "
Nov 02 12:31:15.084005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.083697    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-data" pod "cad06365-a977-4f66-94eb-2511cc1d02f0" (UID: "cad06365-a977-4f66-94eb-2511cc1d02f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:31:15.084005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.083755    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda61e72-8ab1-4390-8295-83a8fc113578-kube-api-access-qctgd\" (UniqueName: \"kubernetes.io/projected/dda61e72-8ab1-4390-8295-83a8fc113578-kube-api-access-qctgd\") pod \"dda61e72-8ab1-4390-8295-83a8fc113578\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") "
Nov 02 12:31:15.084005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.083790    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-istio-data\") pod \"dda61e72-8ab1-4390-8295-83a8fc113578\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") "
Nov 02 12:31:15.084005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.083819    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-workload-certs\") pod \"cad06365-a977-4f66-94eb-2511cc1d02f0\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") "
Nov 02 12:31:15.084005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.083890    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda61e72-8ab1-4390-8295-83a8fc113578-istio-token\" (UniqueName: \"kubernetes.io/projected/dda61e72-8ab1-4390-8295-83a8fc113578-istio-token\") pod \"dda61e72-8ab1-4390-8295-83a8fc113578\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") "
Nov 02 12:31:15.084005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.083923    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-credential-socket" pod "cad06365-a977-4f66-94eb-2511cc1d02f0" (UID: "cad06365-a977-4f66-94eb-2511cc1d02f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:31:15.084005 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.083994    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dda61e72-8ab1-4390-8295-83a8fc113578-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda61e72-8ab1-4390-8295-83a8fc113578-istio-podinfo\") pod \"dda61e72-8ab1-4390-8295-83a8fc113578\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") "
Nov 02 12:31:15.084596 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.084077    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda61e72-8ab1-4390-8295-83a8fc113578-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda61e72-8ab1-4390-8295-83a8fc113578-istiod-ca-cert\") pod \"dda61e72-8ab1-4390-8295-83a8fc113578\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") "
Nov 02 12:31:15.084596 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.084110    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-credential-socket\") pod \"dda61e72-8ab1-4390-8295-83a8fc113578\" (UID: \"dda61e72-8ab1-4390-8295-83a8fc113578\") "
Nov 02 12:31:15.084596 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.084126    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-workload-socket" pod "cad06365-a977-4f66-94eb-2511cc1d02f0" (UID: "cad06365-a977-4f66-94eb-2511cc1d02f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:31:15.084596 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.084140    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-workload-certs" pod "dda61e72-8ab1-4390-8295-83a8fc113578" (UID: "dda61e72-8ab1-4390-8295-83a8fc113578"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:31:15.084596 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.084170    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-envoy\") pod \"cad06365-a977-4f66-94eb-2511cc1d02f0\" (UID: \"cad06365-a977-4f66-94eb-2511cc1d02f0\") "
Nov 02 12:31:15.084596 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.084348    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-workload-socket" pod "dda61e72-8ab1-4390-8295-83a8fc113578" (UID: "dda61e72-8ab1-4390-8295-83a8fc113578"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:31:15.084596 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.084460    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad06365-a977-4f66-94eb-2511cc1d02f0-istiod-ca-cert" pod "cad06365-a977-4f66-94eb-2511cc1d02f0" (UID: "cad06365-a977-4f66-94eb-2511cc1d02f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:31:15.085038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.084684    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.085038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.084708    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.085038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.084723    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad06365-a977-4f66-94eb-2511cc1d02f0-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.085038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.084737    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.085038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.084753    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.085038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.084768    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.085038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.085030    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda61e72-8ab1-4390-8295-83a8fc113578-istiod-ca-cert" pod "dda61e72-8ab1-4390-8295-83a8fc113578" (UID: "dda61e72-8ab1-4390-8295-83a8fc113578"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:31:15.085421 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.085084    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-workload-certs" pod "cad06365-a977-4f66-94eb-2511cc1d02f0" (UID: "cad06365-a977-4f66-94eb-2511cc1d02f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:31:15.086122 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.086093    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-credential-socket" pod "dda61e72-8ab1-4390-8295-83a8fc113578" (UID: "dda61e72-8ab1-4390-8295-83a8fc113578"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:31:15.086122 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.086113    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-istio-data" pod "dda61e72-8ab1-4390-8295-83a8fc113578" (UID: "dda61e72-8ab1-4390-8295-83a8fc113578"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:31:15.086677 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.086641    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-podinfo" pod "cad06365-a977-4f66-94eb-2511cc1d02f0" (UID: "cad06365-a977-4f66-94eb-2511cc1d02f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:31:15.086677 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.086653    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-istio-envoy" pod "dda61e72-8ab1-4390-8295-83a8fc113578" (UID: "dda61e72-8ab1-4390-8295-83a8fc113578"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:31:15.086832 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.086717    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-envoy" pod "cad06365-a977-4f66-94eb-2511cc1d02f0" (UID: "cad06365-a977-4f66-94eb-2511cc1d02f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:31:15.086943 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.086919    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-token" pod "cad06365-a977-4f66-94eb-2511cc1d02f0" (UID: "cad06365-a977-4f66-94eb-2511cc1d02f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:31:15.087049 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.087029    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad06365-a977-4f66-94eb-2511cc1d02f0-kube-api-access-2q5l2" pod "cad06365-a977-4f66-94eb-2511cc1d02f0" (UID: "cad06365-a977-4f66-94eb-2511cc1d02f0"). InnerVolumeSpecName "kube-api-access-2q5l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:31:15.087305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.087288    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dda61e72-8ab1-4390-8295-83a8fc113578-istio-podinfo" pod "dda61e72-8ab1-4390-8295-83a8fc113578" (UID: "dda61e72-8ab1-4390-8295-83a8fc113578"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:31:15.087706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.087688    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda61e72-8ab1-4390-8295-83a8fc113578-kube-api-access-qctgd" pod "dda61e72-8ab1-4390-8295-83a8fc113578" (UID: "dda61e72-8ab1-4390-8295-83a8fc113578"). InnerVolumeSpecName "kube-api-access-qctgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:31:15.087765 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.087705    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda61e72-8ab1-4390-8295-83a8fc113578-istio-token" pod "dda61e72-8ab1-4390-8295-83a8fc113578" (UID: "dda61e72-8ab1-4390-8295-83a8fc113578"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:31:15.186468 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.186389    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda61e72-8ab1-4390-8295-83a8fc113578-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.186468 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.186429    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda61e72-8ab1-4390-8295-83a8fc113578-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.186468 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.186445    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.186468 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.186459    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.186724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.186473    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.186724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.186489    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q5l2\" (UniqueName: \"kubernetes.io/projected/cad06365-a977-4f66-94eb-2511cc1d02f0-kube-api-access-2q5l2\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.186724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.186503    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad06365-a977-4f66-94eb-2511cc1d02f0-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.186724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.186517    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.186724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.186531    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qctgd\" (UniqueName: \"kubernetes.io/projected/dda61e72-8ab1-4390-8295-83a8fc113578-kube-api-access-qctgd\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.186724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.186544    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda61e72-8ab1-4390-8295-83a8fc113578-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.186724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.186563    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad06365-a977-4f66-94eb-2511cc1d02f0-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.186724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.186577    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda61e72-8ab1-4390-8295-83a8fc113578-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:31:15.257940 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.257907    4464 generic.go:356] "Generic (PLEG): container finished" podID="dda61e72-8ab1-4390-8295-83a8fc113578" containerID="d5aa3a11acdd3c1d9a91f07eb8c1c2f3d2ea4c1a9fe5333a29b2eaac8d901ead" exitCode=0
Nov 02 12:31:15.258117 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.257956    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"
Nov 02 12:31:15.258117 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.258011    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p" event={"ID":"dda61e72-8ab1-4390-8295-83a8fc113578","Type":"ContainerDied","Data":"d5aa3a11acdd3c1d9a91f07eb8c1c2f3d2ea4c1a9fe5333a29b2eaac8d901ead"}
Nov 02 12:31:15.258117 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.258065    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p" event={"ID":"dda61e72-8ab1-4390-8295-83a8fc113578","Type":"ContainerDied","Data":"91fe58c626101525e77640b04fb89d7bf262f67b6c94ade3f9f08e11b4cd7bcd"}
Nov 02 12:31:15.258117 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.258093    4464 scope.go:122] "RemoveContainer" containerID="d5aa3a11acdd3c1d9a91f07eb8c1c2f3d2ea4c1a9fe5333a29b2eaac8d901ead"
Nov 02 12:31:15.259394 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.259369    4464 generic.go:356] "Generic (PLEG): container finished" podID="cad06365-a977-4f66-94eb-2511cc1d02f0" containerID="0922e9cf37b9b2fff62d9a673d3e8d456fb5de6c330e8826209f3bd53fc4c443" exitCode=0
Nov 02 12:31:15.259472 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.259417    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"
Nov 02 12:31:15.259519 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.259414    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9" event={"ID":"cad06365-a977-4f66-94eb-2511cc1d02f0","Type":"ContainerDied","Data":"0922e9cf37b9b2fff62d9a673d3e8d456fb5de6c330e8826209f3bd53fc4c443"}
Nov 02 12:31:15.259567 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.259548    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9" event={"ID":"cad06365-a977-4f66-94eb-2511cc1d02f0","Type":"ContainerDied","Data":"eea5e7f129c9c09f0766b81f48bff3b83ba9148010ed1b25af615c1cb0a30694"}
Nov 02 12:31:15.270039 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.269881    4464 scope.go:122] "RemoveContainer" containerID="d5aa3a11acdd3c1d9a91f07eb8c1c2f3d2ea4c1a9fe5333a29b2eaac8d901ead"
Nov 02 12:31:15.270289 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:31:15.270263    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5aa3a11acdd3c1d9a91f07eb8c1c2f3d2ea4c1a9fe5333a29b2eaac8d901ead\": container with ID starting with d5aa3a11acdd3c1d9a91f07eb8c1c2f3d2ea4c1a9fe5333a29b2eaac8d901ead not found: ID does not exist" containerID="d5aa3a11acdd3c1d9a91f07eb8c1c2f3d2ea4c1a9fe5333a29b2eaac8d901ead"
Nov 02 12:31:15.270349 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.270302    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5aa3a11acdd3c1d9a91f07eb8c1c2f3d2ea4c1a9fe5333a29b2eaac8d901ead"} err="failed to get container status \"d5aa3a11acdd3c1d9a91f07eb8c1c2f3d2ea4c1a9fe5333a29b2eaac8d901ead\": rpc error: code = NotFound desc = could not find container \"d5aa3a11acdd3c1d9a91f07eb8c1c2f3d2ea4c1a9fe5333a29b2eaac8d901ead\": container with ID starting with d5aa3a11acdd3c1d9a91f07eb8c1c2f3d2ea4c1a9fe5333a29b2eaac8d901ead not found: ID does not exist"
Nov 02 12:31:15.270349 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.270324    4464 scope.go:122] "RemoveContainer" containerID="0922e9cf37b9b2fff62d9a673d3e8d456fb5de6c330e8826209f3bd53fc4c443"
Nov 02 12:31:15.281786 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.281765    4464 scope.go:122] "RemoveContainer" containerID="0922e9cf37b9b2fff62d9a673d3e8d456fb5de6c330e8826209f3bd53fc4c443"
Nov 02 12:31:15.282106 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:31:15.282089    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0922e9cf37b9b2fff62d9a673d3e8d456fb5de6c330e8826209f3bd53fc4c443\": container with ID starting with 0922e9cf37b9b2fff62d9a673d3e8d456fb5de6c330e8826209f3bd53fc4c443 not found: ID does not exist" containerID="0922e9cf37b9b2fff62d9a673d3e8d456fb5de6c330e8826209f3bd53fc4c443"
Nov 02 12:31:15.282169 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.282115    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0922e9cf37b9b2fff62d9a673d3e8d456fb5de6c330e8826209f3bd53fc4c443"} err="failed to get container status \"0922e9cf37b9b2fff62d9a673d3e8d456fb5de6c330e8826209f3bd53fc4c443\": rpc error: code = NotFound desc = could not find container \"0922e9cf37b9b2fff62d9a673d3e8d456fb5de6c330e8826209f3bd53fc4c443\": container with ID starting with 0922e9cf37b9b2fff62d9a673d3e8d456fb5de6c330e8826209f3bd53fc4c443 not found: ID does not exist"
Nov 02 12:31:15.403387 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.403303    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"]
Nov 02 12:31:15.406949 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.406923    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-9bd6p"]
Nov 02 12:31:15.520896 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.520865    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"]
Nov 02 12:31:15.525179 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.525154    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-lcmx9"]
Nov 02 12:31:15.582361 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.582332    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad06365-a977-4f66-94eb-2511cc1d02f0" path="/var/lib/kubelet/pods/cad06365-a977-4f66-94eb-2511cc1d02f0/volumes"
Nov 02 12:31:15.582807 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.582793    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda61e72-8ab1-4390-8295-83a8fc113578" path="/var/lib/kubelet/pods/dda61e72-8ab1-4390-8295-83a8fc113578/volumes"
Nov 02 12:31:15.951530 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.951452    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"]
Nov 02 12:31:15.955971 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:15.955948    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-02304760-gt9cr"]
Nov 02 12:31:17.519455 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.519411    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"]
Nov 02 12:31:17.545743 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.545709    4464 cpu_manager.go:414] "containerMap: removing container" podUID="d89ec611-1f60-49e5-80b9-ef7720ecb732" containerName="payload-pre-processing"
Nov 02 12:31:17.556095 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.556051    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89ec611-1f60-49e5-80b9-ef7720ecb732" containerName="payload-pre-processing"
Nov 02 12:31:17.556323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.556304    4464 cpu_manager.go:414] "containerMap: removing container" podUID="964e4b45-70c3-4ab2-8960-cf5f56c36a04" containerName="payload-pre-processing"
Nov 02 12:31:17.556457 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.556443    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="964e4b45-70c3-4ab2-8960-cf5f56c36a04" containerName="payload-pre-processing"
Nov 02 12:31:17.556620 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.556605    4464 cpu_manager.go:414] "containerMap: removing container" podUID="8eb017b2-26e3-450a-953f-e348a1109d20" containerName="main"
Nov 02 12:31:17.557029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.557006    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb017b2-26e3-450a-953f-e348a1109d20" containerName="main"
Nov 02 12:31:17.557178 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.557166    4464 cpu_manager.go:414] "containerMap: removing container" podUID="cad06365-a977-4f66-94eb-2511cc1d02f0" containerName="istio-proxy"
Nov 02 12:31:17.557283 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.557271    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad06365-a977-4f66-94eb-2511cc1d02f0" containerName="istio-proxy"
Nov 02 12:31:17.557377 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.557367    4464 cpu_manager.go:414] "containerMap: removing container" podUID="c1e079e8-6acf-41b9-b8a8-1a066acccb93" containerName="payload-processing"
Nov 02 12:31:17.557466 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.557456    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e079e8-6acf-41b9-b8a8-1a066acccb93" containerName="payload-processing"
Nov 02 12:31:17.557590 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.557577    4464 cpu_manager.go:414] "containerMap: removing container" podUID="dda61e72-8ab1-4390-8295-83a8fc113578" containerName="istio-proxy"
Nov 02 12:31:17.557672 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.557661    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda61e72-8ab1-4390-8295-83a8fc113578" containerName="istio-proxy"
Nov 02 12:31:17.557773 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.557762    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e85c3283-ce74-4e7f-b191-79e497d42a77" containerName="payload-processing"
Nov 02 12:31:17.557855 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.557844    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e85c3283-ce74-4e7f-b191-79e497d42a77" containerName="payload-processing"
Nov 02 12:31:17.558953 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.558933    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e079e8-6acf-41b9-b8a8-1a066acccb93" containerName="payload-processing"
Nov 02 12:31:17.559117 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.559105    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda61e72-8ab1-4390-8295-83a8fc113578" containerName="istio-proxy"
Nov 02 12:31:17.559227 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.559216    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e85c3283-ce74-4e7f-b191-79e497d42a77" containerName="payload-processing"
Nov 02 12:31:17.559321 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.559312    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89ec611-1f60-49e5-80b9-ef7720ecb732" containerName="payload-pre-processing"
Nov 02 12:31:17.559406 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.559394    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="964e4b45-70c3-4ab2-8960-cf5f56c36a04" containerName="payload-pre-processing"
Nov 02 12:31:17.559493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.559484    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb017b2-26e3-450a-953f-e348a1109d20" containerName="main"
Nov 02 12:31:17.559562 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.559553    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad06365-a977-4f66-94eb-2511cc1d02f0" containerName="istio-proxy"
Nov 02 12:31:17.571715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.571686    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.601615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.595453    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="503f6757-3f34-4003-9559-22244e0d3283" path="/var/lib/kubelet/pods/503f6757-3f34-4003-9559-22244e0d3283/volumes"
Nov 02 12:31:17.601615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.595886    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"]
Nov 02 12:31:17.648025 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.639304    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2e101a-openshift-default-dockercfg-knf8t\""
Nov 02 12:31:17.672535 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.672497    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-podinfo\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.672857 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.672841    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-data\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.673044 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.673030    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbh6\" (UniqueName: \"kubernetes.io/projected/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-kube-api-access-sgbh6\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.673212 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.673181    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-token\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.673407 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.673391    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-credential-socket\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.674611 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.674587    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-envoy\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.674848 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.674832    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-workload-certs\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.675050 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.675035    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-workload-socket\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.675261 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.675203    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istiod-ca-cert\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.776493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.776416    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-workload-certs\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.776493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.776463    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-workload-socket\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.776683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.776491    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istiod-ca-cert\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.776683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.776541    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-podinfo\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.776683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.776577    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-data\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.776683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.776622    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbh6\" (UniqueName: \"kubernetes.io/projected/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-kube-api-access-sgbh6\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.776683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.776650    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-token\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.776868 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.776710    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-credential-socket\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.776868 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.776827    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-envoy\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.776868 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.776859    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-workload-certs\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.777009 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.776929    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-workload-socket\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.777009 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.776962    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-data\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.777240 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.777210    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-credential-socket\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.777373 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.777265    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istiod-ca-cert\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.778955 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.778933    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-envoy\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.779189 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.779171    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-podinfo\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.841357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.841323    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-token\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.842711 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.842684    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbh6\" (UniqueName: \"kubernetes.io/projected/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-kube-api-access-sgbh6\") pod \"e2e-api-a-2e101a-openshift-default-88548686f-drm8q\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") " pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:17.906149 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:17.906122    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:18.340561 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:31:18.340534    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d041eb_5ae8_46ea_b1cc_cebfa1973a1e.slice/crio-b538bf495c86c263a1cda1ecf658c324c4cec18ae0f64fdabce817377c16690e WatchSource:0}: Error finding container b538bf495c86c263a1cda1ecf658c324c4cec18ae0f64fdabce817377c16690e: Status 404 returned error can't find the container with id b538bf495c86c263a1cda1ecf658c324c4cec18ae0f64fdabce817377c16690e
Nov 02 12:31:18.341944 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:18.341910    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"]
Nov 02 12:31:18.343638 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:18.343607    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:31:18.343706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:18.343673    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:31:18.343706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:18.343700    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:31:19.290489 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:19.290455    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q" event={"ID":"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e","Type":"ContainerStarted","Data":"cdf627f5ce63cf92589fd95c214f9c3a27a04658d89262ebe91bb9d763249a51"}
Nov 02 12:31:19.290957 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:19.290518    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q" event={"ID":"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e","Type":"ContainerStarted","Data":"b538bf495c86c263a1cda1ecf658c324c4cec18ae0f64fdabce817377c16690e"}
Nov 02 12:31:19.466539 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:19.466485    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q" podStartSLOduration=2.466471549 podStartE2EDuration="2.466471549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:31:19.463853832 +0000 UTC m=+2108.027562244" watchObservedRunningTime="2026-11-02 12:31:19.466471549 +0000 UTC m=+2108.030179957"
Nov 02 12:31:19.906399 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:19.906369    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:19.911924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:19.911900    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:20.298728 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:20.298681    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:20.300113 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:20.300087    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:31:26.410358 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:26.410256    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"]
Nov 02 12:31:26.437092 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:26.437057    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"]
Nov 02 12:31:26.437277 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:26.437180    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"
Nov 02 12:31:26.517547 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:26.517514    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2e101a-dockercfg-npbrw\""
Nov 02 12:31:26.517547 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:26.517514    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2e101a\""
Nov 02 12:31:26.520232 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:26.520211    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc6wk\" (UniqueName: \"kubernetes.io/projected/fcfaa90b-9297-4cfe-b835-29d2454322ac-kube-api-access-rc6wk\") pod \"payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v\" (UID: \"fcfaa90b-9297-4cfe-b835-29d2454322ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"
Nov 02 12:31:26.520301 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:26.520248    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfaa90b-9297-4cfe-b835-29d2454322ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v\" (UID: \"fcfaa90b-9297-4cfe-b835-29d2454322ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"
Nov 02 12:31:26.621486 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:26.621449    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6wk\" (UniqueName: \"kubernetes.io/projected/fcfaa90b-9297-4cfe-b835-29d2454322ac-kube-api-access-rc6wk\") pod \"payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v\" (UID: \"fcfaa90b-9297-4cfe-b835-29d2454322ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"
Nov 02 12:31:26.621678 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:26.621512    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfaa90b-9297-4cfe-b835-29d2454322ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v\" (UID: \"fcfaa90b-9297-4cfe-b835-29d2454322ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"
Nov 02 12:31:26.622388 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:26.622366    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfaa90b-9297-4cfe-b835-29d2454322ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v\" (UID: \"fcfaa90b-9297-4cfe-b835-29d2454322ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"
Nov 02 12:31:26.683387 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:26.683312    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6wk\" (UniqueName: \"kubernetes.io/projected/fcfaa90b-9297-4cfe-b835-29d2454322ac-kube-api-access-rc6wk\") pod \"payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v\" (UID: \"fcfaa90b-9297-4cfe-b835-29d2454322ac\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"
Nov 02 12:31:26.753623 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:26.753587    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"
Nov 02 12:31:27.129319 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:31:27.129281    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcfaa90b_9297_4cfe_b835_29d2454322ac.slice/crio-56be09a043534002bff16fbaaa2f2445b48a432c88e40e1edd3f2a4842c18846 WatchSource:0}: Error finding container 56be09a043534002bff16fbaaa2f2445b48a432c88e40e1edd3f2a4842c18846: Status 404 returned error can't find the container with id 56be09a043534002bff16fbaaa2f2445b48a432c88e40e1edd3f2a4842c18846
Nov 02 12:31:27.129468 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:27.129412    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"]
Nov 02 12:31:27.355193 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:27.355159    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v" event={"ID":"fcfaa90b-9297-4cfe-b835-29d2454322ac","Type":"ContainerStarted","Data":"c9342fe66fd039c2d55d437abc3acf94d8743b152a4a9c4a60371227ca95ef5e"}
Nov 02 12:31:27.355390 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:27.355208    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v" event={"ID":"fcfaa90b-9297-4cfe-b835-29d2454322ac","Type":"ContainerStarted","Data":"56be09a043534002bff16fbaaa2f2445b48a432c88e40e1edd3f2a4842c18846"}
Nov 02 12:31:27.355390 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:27.355282    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"
Nov 02 12:31:27.510001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:27.505252    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v" podStartSLOduration=1.505234308 podStartE2EDuration="1.505234308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:31:27.501577924 +0000 UTC m=+2116.065286334" watchObservedRunningTime="2026-11-02 12:31:27.505234308 +0000 UTC m=+2116.068942719"
Nov 02 12:31:28.293362 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:31:28.293330    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:31:36.761203 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:36.761163    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:31:38.390575 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:38.390536    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"
Nov 02 12:31:46.523317 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:46.523284    4464 generic.go:356] "Generic (PLEG): container finished" podID="a7a7742a-f60b-4270-8992-2855b79bed42" containerID="f81cb9696d14ff74d92bc0b637701470e44d76acca6b095b50336d6f61106064" exitCode=0
Nov 02 12:31:46.523849 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:46.523356    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j" event={"ID":"a7a7742a-f60b-4270-8992-2855b79bed42","Type":"ContainerDied","Data":"f81cb9696d14ff74d92bc0b637701470e44d76acca6b095b50336d6f61106064"}
Nov 02 12:31:46.523849 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:46.523843    4464 scope.go:122] "RemoveContainer" containerID="f81cb9696d14ff74d92bc0b637701470e44d76acca6b095b50336d6f61106064"
Nov 02 12:31:46.525250 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:46.525224    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:31:46.525341 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:46.525288    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:31:46.525341 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:46.525319    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:31:47.532541 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:47.532508    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j" event={"ID":"a7a7742a-f60b-4270-8992-2855b79bed42","Type":"ContainerStarted","Data":"f6604e285d546b6f3d4319328b477b84071373fbb490ef3118e000161cb59194"}
Nov 02 12:31:47.895550 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:47.895443    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:31:48.435325 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:48.435291    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:31:48.436391 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:48.436365    4464 patch_prober.go:29] interesting pod/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 02 12:31:48.436518 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:48.436435    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j" podUID="a7a7742a-f60b-4270-8992-2855b79bed42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 02 12:31:49.435484 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:49.435454    4464 patch_prober.go:29] interesting pod/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 02 12:31:49.435867 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:49.435505    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j" podUID="a7a7742a-f60b-4270-8992-2855b79bed42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 02 12:31:50.435496 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:50.435460    4464 patch_prober.go:29] interesting pod/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 02 12:31:50.435892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:50.435516    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j" podUID="a7a7742a-f60b-4270-8992-2855b79bed42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 02 12:31:51.436008 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:51.435952    4464 patch_prober.go:29] interesting pod/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 02 12:31:51.436394 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:51.436018    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j" podUID="a7a7742a-f60b-4270-8992-2855b79bed42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 02 12:31:52.435607 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:52.435572    4464 patch_prober.go:29] interesting pod/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 02 12:31:52.435792 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:52.435628    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j" podUID="a7a7742a-f60b-4270-8992-2855b79bed42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 02 12:31:53.439728 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:53.439693    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:31:53.440731 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:31:53.440713    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:31:58.291672 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:31:58.291631    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:32:07.179687 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:07.179648    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:32:16.049038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:16.049004    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"]
Nov 02 12:32:16.099116 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:16.099079    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eba33e27-jwkkz"]
Nov 02 12:32:16.763037 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:16.763001    4464 generic.go:356] "Generic (PLEG): container finished" podID="51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" containerID="cdf627f5ce63cf92589fd95c214f9c3a27a04658d89262ebe91bb9d763249a51" exitCode=0
Nov 02 12:32:16.763269 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:16.763061    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q" event={"ID":"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e","Type":"ContainerDied","Data":"cdf627f5ce63cf92589fd95c214f9c3a27a04658d89262ebe91bb9d763249a51"}
Nov 02 12:32:16.763484 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:16.763469    4464 scope.go:122] "RemoveContainer" containerID="cdf627f5ce63cf92589fd95c214f9c3a27a04658d89262ebe91bb9d763249a51"
Nov 02 12:32:16.764837 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:16.764811    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:32:16.764888 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:16.764876    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:32:16.764923 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:16.764904    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:32:17.589193 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:17.589161    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc45a848-4783-4184-aab0-965d9e164047" path="/var/lib/kubelet/pods/cc45a848-4783-4184-aab0-965d9e164047/volumes"
Nov 02 12:32:17.790719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:17.790679    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q" event={"ID":"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e","Type":"ContainerStarted","Data":"c3d50eacb69a9f21ec1d3260fd170832d0f2af2e1d1600855d97901276a38072"}
Nov 02 12:32:17.907309 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:17.907222    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:32:17.908504 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:17.908481    4464 patch_prober.go:29] interesting pod/e2e-api-a-2e101a-openshift-default-88548686f-drm8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 12:32:17.908563 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:17.908528    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q" podUID="51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 12:32:18.906603 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:18.906566    4464 patch_prober.go:29] interesting pod/e2e-api-a-2e101a-openshift-default-88548686f-drm8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 12:32:18.907073 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:18.906618    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q" podUID="51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 12:32:19.906813 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:19.906761    4464 patch_prober.go:29] interesting pod/e2e-api-a-2e101a-openshift-default-88548686f-drm8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 12:32:19.907241 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:19.906826    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q" podUID="51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 12:32:20.299268 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:20.299227    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:32:20.906585 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:20.906542    4464 patch_prober.go:29] interesting pod/e2e-api-a-2e101a-openshift-default-88548686f-drm8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 12:32:20.906775 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:20.906603    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q" podUID="51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 12:32:21.906649 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:21.906615    4464 patch_prober.go:29] interesting pod/e2e-api-a-2e101a-openshift-default-88548686f-drm8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 12:32:21.907043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:21.906664    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q" podUID="51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 12:32:22.907264 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:22.907226    4464 patch_prober.go:29] interesting pod/e2e-api-a-2e101a-openshift-default-88548686f-drm8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 02 12:32:22.907671 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:22.907292    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q" podUID="51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 02 12:32:23.049016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:23.048971    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"]
Nov 02 12:32:23.073779 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:23.073754    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfe8bfdd-t6pbs"]
Nov 02 12:32:23.581965 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:23.581931    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3463300d-10e1-4637-b554-e9f3a34f9a6b" path="/var/lib/kubelet/pods/3463300d-10e1-4637-b554-e9f3a34f9a6b/volumes"
Nov 02 12:32:23.910438 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:23.910362    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:32:23.911682 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:23.911662    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:32:28.292216 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:32:28.292177    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:32:28.777985 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:28.777945    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:32:34.051991 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:34.051939    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"]
Nov 02 12:32:34.102361 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:34.102337    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7d1db0-vjnmz"]
Nov 02 12:32:35.048031 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:35.047998    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"]
Nov 02 12:32:35.098306 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:35.098277    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f841c1ed-mnnv5"]
Nov 02 12:32:35.582342 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:35.582308    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="026d0a04-126f-4959-94a2-9300876e2bb1" path="/var/lib/kubelet/pods/026d0a04-126f-4959-94a2-9300876e2bb1/volumes"
Nov 02 12:32:35.582771 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:35.582758    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1" path="/var/lib/kubelet/pods/60ffd0dc-fcde-48f8-8f53-3283f0f0dfb1/volumes"
Nov 02 12:32:48.047515 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:48.047477    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"]
Nov 02 12:32:48.089667 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:48.089641    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3c97f6c-z2nrg"]
Nov 02 12:32:49.582031 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:49.581999    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f82fe7-86f1-494b-ba79-dbec14b80cc0" path="/var/lib/kubelet/pods/f7f82fe7-86f1-494b-ba79-dbec14b80cc0/volumes"
Nov 02 12:32:53.717998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:53.713921    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"]
Nov 02 12:32:53.744317 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:53.744284    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"
Nov 02 12:32:53.770686 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:53.770652    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"]
Nov 02 12:32:53.812779 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:53.812748    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-257e6d\""
Nov 02 12:32:53.813006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:53.812836    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-257e6d-dockercfg-s579g\""
Nov 02 12:32:53.870362 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:53.870325    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-257e6d-86977c947f-t9chn\" (UID: \"c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5\") " pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"
Nov 02 12:32:53.870506 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:53.870439    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjdtn\" (UniqueName: \"kubernetes.io/projected/c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5-kube-api-access-hjdtn\") pod \"payload-processing-e2e-api-b-257e6d-86977c947f-t9chn\" (UID: \"c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5\") " pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"
Nov 02 12:32:53.971516 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:53.971486    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-257e6d-86977c947f-t9chn\" (UID: \"c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5\") " pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"
Nov 02 12:32:53.971678 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:53.971548    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjdtn\" (UniqueName: \"kubernetes.io/projected/c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5-kube-api-access-hjdtn\") pod \"payload-processing-e2e-api-b-257e6d-86977c947f-t9chn\" (UID: \"c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5\") " pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"
Nov 02 12:32:53.972229 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:53.972204    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-257e6d-86977c947f-t9chn\" (UID: \"c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5\") " pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"
Nov 02 12:32:54.033489 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:54.033461    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjdtn\" (UniqueName: \"kubernetes.io/projected/c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5-kube-api-access-hjdtn\") pod \"payload-processing-e2e-api-b-257e6d-86977c947f-t9chn\" (UID: \"c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5\") " pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"
Nov 02 12:32:54.071273 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:54.071244    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"
Nov 02 12:32:54.456028 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:54.455998    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"]
Nov 02 12:32:54.456200 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:32:54.456178    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56fd4ed_661a_4a95_b6cb_062ff2aa2ca5.slice/crio-41dd71b6466aa9299fdf74bbee891a3df7dc06c90c3963d2ceda3cbc0651a108 WatchSource:0}: Error finding container 41dd71b6466aa9299fdf74bbee891a3df7dc06c90c3963d2ceda3cbc0651a108: Status 404 returned error can't find the container with id 41dd71b6466aa9299fdf74bbee891a3df7dc06c90c3963d2ceda3cbc0651a108
Nov 02 12:32:55.061398 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:55.061362    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn" event={"ID":"c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5","Type":"ContainerStarted","Data":"05f0a367b9b1faab6c898c6f8508516b096fc77e87650355ded89fc18364d2aa"}
Nov 02 12:32:55.061832 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:55.061444    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn" event={"ID":"c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5","Type":"ContainerStarted","Data":"41dd71b6466aa9299fdf74bbee891a3df7dc06c90c3963d2ceda3cbc0651a108"}
Nov 02 12:32:55.061832 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:55.061512    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"
Nov 02 12:32:55.179240 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:55.179191    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn" podStartSLOduration=2.179178562 podStartE2EDuration="2.179178562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:32:55.178223974 +0000 UTC m=+2203.741932383" watchObservedRunningTime="2026-11-02 12:32:55.179178562 +0000 UTC m=+2203.742886970"
Nov 02 12:32:56.919216 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:56.919183    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"]
Nov 02 12:32:56.922320 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:56.922304    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"
Nov 02 12:32:57.006362 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:57.006324    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p5jf\" (UniqueName: \"kubernetes.io/projected/17ec1103-8501-449e-9892-2a1cd83aa9b0-kube-api-access-7p5jf\") pod \"maas-api-revoke-sub-aa771475-hclwn\" (UID: \"17ec1103-8501-449e-9892-2a1cd83aa9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"
Nov 02 12:32:57.006524 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:57.006481    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ec1103-8501-449e-9892-2a1cd83aa9b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa771475-hclwn\" (UID: \"17ec1103-8501-449e-9892-2a1cd83aa9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"
Nov 02 12:32:57.006580 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:57.006531    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ec1103-8501-449e-9892-2a1cd83aa9b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa771475-hclwn\" (UID: \"17ec1103-8501-449e-9892-2a1cd83aa9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"
Nov 02 12:32:57.027199 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:57.027045    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:32:57.027199 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:57.027086    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:32:57.027199 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:57.027118    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:32:57.027199 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:57.027143    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:32:57.030607 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:57.030586    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"]
Nov 02 12:32:57.107786 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:57.107762    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p5jf\" (UniqueName: \"kubernetes.io/projected/17ec1103-8501-449e-9892-2a1cd83aa9b0-kube-api-access-7p5jf\") pod \"maas-api-revoke-sub-aa771475-hclwn\" (UID: \"17ec1103-8501-449e-9892-2a1cd83aa9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"
Nov 02 12:32:57.107925 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:57.107830    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ec1103-8501-449e-9892-2a1cd83aa9b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa771475-hclwn\" (UID: \"17ec1103-8501-449e-9892-2a1cd83aa9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"
Nov 02 12:32:57.107925 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:57.107861    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ec1103-8501-449e-9892-2a1cd83aa9b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa771475-hclwn\" (UID: \"17ec1103-8501-449e-9892-2a1cd83aa9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"
Nov 02 12:32:57.108533 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:57.108509    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ec1103-8501-449e-9892-2a1cd83aa9b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa771475-hclwn\" (UID: \"17ec1103-8501-449e-9892-2a1cd83aa9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"
Nov 02 12:32:57.110173 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:57.110150    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ec1103-8501-449e-9892-2a1cd83aa9b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa771475-hclwn\" (UID: \"17ec1103-8501-449e-9892-2a1cd83aa9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"
Nov 02 12:32:57.169553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:57.169499    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p5jf\" (UniqueName: \"kubernetes.io/projected/17ec1103-8501-449e-9892-2a1cd83aa9b0-kube-api-access-7p5jf\") pod \"maas-api-revoke-sub-aa771475-hclwn\" (UID: \"17ec1103-8501-449e-9892-2a1cd83aa9b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"
Nov 02 12:32:57.234671 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:57.234641    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"
Nov 02 12:32:57.605931 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:57.605902    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"]
Nov 02 12:32:57.606775 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:32:57.606751    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ec1103_8501_449e_9892_2a1cd83aa9b0.slice/crio-cf067ac92156659bab6d4a15e0bc630402448f0546fac25b8e49f0f7e068f946 WatchSource:0}: Error finding container cf067ac92156659bab6d4a15e0bc630402448f0546fac25b8e49f0f7e068f946: Status 404 returned error can't find the container with id cf067ac92156659bab6d4a15e0bc630402448f0546fac25b8e49f0f7e068f946
Nov 02 12:32:58.087217 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:58.087181    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn" event={"ID":"17ec1103-8501-449e-9892-2a1cd83aa9b0","Type":"ContainerStarted","Data":"51073f63168072b4161f1eca3cfdd26735e9d97135d598628c5ac20dd5ea3e96"}
Nov 02 12:32:58.087597 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:58.087237    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn" event={"ID":"17ec1103-8501-449e-9892-2a1cd83aa9b0","Type":"ContainerStarted","Data":"cf067ac92156659bab6d4a15e0bc630402448f0546fac25b8e49f0f7e068f946"}
Nov 02 12:32:58.208420 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:58.208332    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn" podStartSLOduration=2.208318861 podStartE2EDuration="2.208318861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:32:58.204887838 +0000 UTC m=+2206.768596261" watchObservedRunningTime="2026-11-02 12:32:58.208318861 +0000 UTC m=+2206.772027270"
Nov 02 12:32:58.292330 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:32:58.292298    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:32:59.047901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:59.047865    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"]
Nov 02 12:32:59.106853 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:59.106823    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc4c8313-swfdf"]
Nov 02 12:32:59.107286 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:59.107206    4464 generic.go:356] "Generic (PLEG): container finished" podID="17ec1103-8501-449e-9892-2a1cd83aa9b0" containerID="51073f63168072b4161f1eca3cfdd26735e9d97135d598628c5ac20dd5ea3e96" exitCode=0
Nov 02 12:32:59.107557 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:59.107306    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn" event={"ID":"17ec1103-8501-449e-9892-2a1cd83aa9b0","Type":"ContainerDied","Data":"51073f63168072b4161f1eca3cfdd26735e9d97135d598628c5ac20dd5ea3e96"}
Nov 02 12:32:59.583329 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:32:59.583298    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97721af5-a292-479f-b254-e989912ebace" path="/var/lib/kubelet/pods/97721af5-a292-479f-b254-e989912ebace/volumes"
Nov 02 12:33:00.263448 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:00.263423    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"
Nov 02 12:33:00.341941 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:00.341914    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17ec1103-8501-449e-9892-2a1cd83aa9b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ec1103-8501-449e-9892-2a1cd83aa9b0-maas-api-xks-ca\") pod \"17ec1103-8501-449e-9892-2a1cd83aa9b0\" (UID: \"17ec1103-8501-449e-9892-2a1cd83aa9b0\") "
Nov 02 12:33:00.342125 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:00.342109    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ec1103-8501-449e-9892-2a1cd83aa9b0-kube-api-access-7p5jf\" (UniqueName: \"kubernetes.io/projected/17ec1103-8501-449e-9892-2a1cd83aa9b0-kube-api-access-7p5jf\") pod \"17ec1103-8501-449e-9892-2a1cd83aa9b0\" (UID: \"17ec1103-8501-449e-9892-2a1cd83aa9b0\") "
Nov 02 12:33:00.342244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:00.342226    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17ec1103-8501-449e-9892-2a1cd83aa9b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ec1103-8501-449e-9892-2a1cd83aa9b0-maas-api-openshift-ca\") pod \"17ec1103-8501-449e-9892-2a1cd83aa9b0\" (UID: \"17ec1103-8501-449e-9892-2a1cd83aa9b0\") "
Nov 02 12:33:00.342550 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:00.342524    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ec1103-8501-449e-9892-2a1cd83aa9b0-maas-api-openshift-ca" pod "17ec1103-8501-449e-9892-2a1cd83aa9b0" (UID: "17ec1103-8501-449e-9892-2a1cd83aa9b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:33:00.342763 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:00.342747    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17ec1103-8501-449e-9892-2a1cd83aa9b0-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:00.343929 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:00.343905    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ec1103-8501-449e-9892-2a1cd83aa9b0-maas-api-xks-ca" pod "17ec1103-8501-449e-9892-2a1cd83aa9b0" (UID: "17ec1103-8501-449e-9892-2a1cd83aa9b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:33:00.344227 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:00.344206    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ec1103-8501-449e-9892-2a1cd83aa9b0-kube-api-access-7p5jf" pod "17ec1103-8501-449e-9892-2a1cd83aa9b0" (UID: "17ec1103-8501-449e-9892-2a1cd83aa9b0"). InnerVolumeSpecName "kube-api-access-7p5jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:33:00.444475 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:00.444403    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17ec1103-8501-449e-9892-2a1cd83aa9b0-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:00.444475 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:00.444432    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p5jf\" (UniqueName: \"kubernetes.io/projected/17ec1103-8501-449e-9892-2a1cd83aa9b0-kube-api-access-7p5jf\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:01.121611 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:01.121581    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"
Nov 02 12:33:01.121781 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:01.121616    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn" event={"ID":"17ec1103-8501-449e-9892-2a1cd83aa9b0","Type":"ContainerDied","Data":"cf067ac92156659bab6d4a15e0bc630402448f0546fac25b8e49f0f7e068f946"}
Nov 02 12:33:01.121781 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:01.121665    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf067ac92156659bab6d4a15e0bc630402448f0546fac25b8e49f0f7e068f946"
Nov 02 12:33:01.287561 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:01.287532    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"]
Nov 02 12:33:06.071609 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:06.071581    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"
Nov 02 12:33:09.875887 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:09.875798    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"]
Nov 02 12:33:09.876594 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:09.876573    4464 cpu_manager.go:414] "containerMap: removing container" podUID="17ec1103-8501-449e-9892-2a1cd83aa9b0" containerName="revoke-keys"
Nov 02 12:33:09.876712 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:09.876596    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ec1103-8501-449e-9892-2a1cd83aa9b0" containerName="revoke-keys"
Nov 02 12:33:09.876825 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:09.876811    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ec1103-8501-449e-9892-2a1cd83aa9b0" containerName="revoke-keys"
Nov 02 12:33:09.880343 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:09.880322    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"
Nov 02 12:33:09.939700 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:09.938577    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:33:09.975161 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:09.975119    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgpjk\" (UniqueName: \"kubernetes.io/projected/bb7ab1ff-866c-4d73-bccb-7eded744c791-kube-api-access-mgpjk\") pod \"maas-api-revoke-sub-a9aabe51-dpt82\" (UID: \"bb7ab1ff-866c-4d73-bccb-7eded744c791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"
Nov 02 12:33:09.975373 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:09.975198    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb7ab1ff-866c-4d73-bccb-7eded744c791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9aabe51-dpt82\" (UID: \"bb7ab1ff-866c-4d73-bccb-7eded744c791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"
Nov 02 12:33:09.975373 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:09.975304    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb7ab1ff-866c-4d73-bccb-7eded744c791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9aabe51-dpt82\" (UID: \"bb7ab1ff-866c-4d73-bccb-7eded744c791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"
Nov 02 12:33:09.988966 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:09.988934    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:33:09.989146 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:09.989054    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:33:09.989146 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:09.989139    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:33:09.992720 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:09.992694    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"]
Nov 02 12:33:10.076358 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:10.076313    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb7ab1ff-866c-4d73-bccb-7eded744c791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9aabe51-dpt82\" (UID: \"bb7ab1ff-866c-4d73-bccb-7eded744c791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"
Nov 02 12:33:10.076571 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:10.076547    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgpjk\" (UniqueName: \"kubernetes.io/projected/bb7ab1ff-866c-4d73-bccb-7eded744c791-kube-api-access-mgpjk\") pod \"maas-api-revoke-sub-a9aabe51-dpt82\" (UID: \"bb7ab1ff-866c-4d73-bccb-7eded744c791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"
Nov 02 12:33:10.076648 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:10.076606    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb7ab1ff-866c-4d73-bccb-7eded744c791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9aabe51-dpt82\" (UID: \"bb7ab1ff-866c-4d73-bccb-7eded744c791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"
Nov 02 12:33:10.077453 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:10.077423    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb7ab1ff-866c-4d73-bccb-7eded744c791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9aabe51-dpt82\" (UID: \"bb7ab1ff-866c-4d73-bccb-7eded744c791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"
Nov 02 12:33:10.078920 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:10.078896    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb7ab1ff-866c-4d73-bccb-7eded744c791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9aabe51-dpt82\" (UID: \"bb7ab1ff-866c-4d73-bccb-7eded744c791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"
Nov 02 12:33:10.138472 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:10.138414    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgpjk\" (UniqueName: \"kubernetes.io/projected/bb7ab1ff-866c-4d73-bccb-7eded744c791-kube-api-access-mgpjk\") pod \"maas-api-revoke-sub-a9aabe51-dpt82\" (UID: \"bb7ab1ff-866c-4d73-bccb-7eded744c791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"
Nov 02 12:33:10.194534 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:10.194497    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"
Nov 02 12:33:10.276213 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:10.268684    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"]
Nov 02 12:33:10.380847 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:10.380559    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"]
Nov 02 12:33:10.599897 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:10.583965    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"]
Nov 02 12:33:11.215948 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:11.215908    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82" event={"ID":"bb7ab1ff-866c-4d73-bccb-7eded744c791","Type":"ContainerStarted","Data":"5587c946fad483f440f7f4ccc94254ed2a658671f453c373d06e24b90fbd7b7c"}
Nov 02 12:33:11.216402 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:11.215971    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82" event={"ID":"bb7ab1ff-866c-4d73-bccb-7eded744c791","Type":"ContainerStarted","Data":"e216331c320cddb4199e729bf47050519bc9d51f7e43632737776b1827d19f69"}
Nov 02 12:33:11.334965 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:11.334864    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82" podStartSLOduration=2.33485116 podStartE2EDuration="2.33485116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:33:11.333129485 +0000 UTC m=+2219.896837893" watchObservedRunningTime="2026-11-02 12:33:11.33485116 +0000 UTC m=+2219.898559568"
Nov 02 12:33:12.223704 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:12.223670    4464 generic.go:356] "Generic (PLEG): container finished" podID="bb7ab1ff-866c-4d73-bccb-7eded744c791" containerID="5587c946fad483f440f7f4ccc94254ed2a658671f453c373d06e24b90fbd7b7c" exitCode=0
Nov 02 12:33:12.224104 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:12.223752    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82" event={"ID":"bb7ab1ff-866c-4d73-bccb-7eded744c791","Type":"ContainerDied","Data":"5587c946fad483f440f7f4ccc94254ed2a658671f453c373d06e24b90fbd7b7c"}
Nov 02 12:33:12.459183 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:12.459142    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:33:13.383626 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:13.383603    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"
Nov 02 12:33:13.441336 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:13.441304    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb7ab1ff-866c-4d73-bccb-7eded744c791-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb7ab1ff-866c-4d73-bccb-7eded744c791-maas-api-openshift-ca\") pod \"bb7ab1ff-866c-4d73-bccb-7eded744c791\" (UID: \"bb7ab1ff-866c-4d73-bccb-7eded744c791\") "
Nov 02 12:33:13.441506 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:13.441363    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb7ab1ff-866c-4d73-bccb-7eded744c791-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb7ab1ff-866c-4d73-bccb-7eded744c791-maas-api-xks-ca\") pod \"bb7ab1ff-866c-4d73-bccb-7eded744c791\" (UID: \"bb7ab1ff-866c-4d73-bccb-7eded744c791\") "
Nov 02 12:33:13.441559 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:13.441499    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7ab1ff-866c-4d73-bccb-7eded744c791-kube-api-access-mgpjk\" (UniqueName: \"kubernetes.io/projected/bb7ab1ff-866c-4d73-bccb-7eded744c791-kube-api-access-mgpjk\") pod \"bb7ab1ff-866c-4d73-bccb-7eded744c791\" (UID: \"bb7ab1ff-866c-4d73-bccb-7eded744c791\") "
Nov 02 12:33:13.441757 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:13.441725    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7ab1ff-866c-4d73-bccb-7eded744c791-maas-api-openshift-ca" pod "bb7ab1ff-866c-4d73-bccb-7eded744c791" (UID: "bb7ab1ff-866c-4d73-bccb-7eded744c791"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:33:13.441945 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:13.441928    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb7ab1ff-866c-4d73-bccb-7eded744c791-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:13.443543 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:13.443516    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb7ab1ff-866c-4d73-bccb-7eded744c791-maas-api-xks-ca" pod "bb7ab1ff-866c-4d73-bccb-7eded744c791" (UID: "bb7ab1ff-866c-4d73-bccb-7eded744c791"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:33:13.443636 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:13.443593    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7ab1ff-866c-4d73-bccb-7eded744c791-kube-api-access-mgpjk" pod "bb7ab1ff-866c-4d73-bccb-7eded744c791" (UID: "bb7ab1ff-866c-4d73-bccb-7eded744c791"). InnerVolumeSpecName "kube-api-access-mgpjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:33:13.471648 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:13.471624    4464 scope.go:122] "RemoveContainer" containerID="3913346a456238d019b0f309bac91d898e29d710b6e46b815dc4c5a146738623"
Nov 02 12:33:13.481565 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:13.481545    4464 scope.go:122] "RemoveContainer" containerID="a398e23b1c536ac06573e4d794e0ee1a67683f5fcb1aaaea7162c95156ee7d01"
Nov 02 12:33:13.542852 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:13.542820    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb7ab1ff-866c-4d73-bccb-7eded744c791-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:13.542852 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:13.542845    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgpjk\" (UniqueName: \"kubernetes.io/projected/bb7ab1ff-866c-4d73-bccb-7eded744c791-kube-api-access-mgpjk\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:14.237455 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:14.237422    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"
Nov 02 12:33:14.237455 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:14.237446    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82" event={"ID":"bb7ab1ff-866c-4d73-bccb-7eded744c791","Type":"ContainerDied","Data":"e216331c320cddb4199e729bf47050519bc9d51f7e43632737776b1827d19f69"}
Nov 02 12:33:14.237666 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:14.237500    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e216331c320cddb4199e729bf47050519bc9d51f7e43632737776b1827d19f69"
Nov 02 12:33:14.798295 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:33:14.798263    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ab1ff_866c_4d73_bccb_7eded744c791.slice/crio-conmon-5587c946fad483f440f7f4ccc94254ed2a658671f453c373d06e24b90fbd7b7c.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:33:15.269840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:15.269777    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr" podUID="1b98a955-089d-4f50-a719-4918f7f5a2c0" containerName="payload-processing" containerID="cri-o://3da45036038f9218a498bce32525296e6c6966b93c83900dcbf0cfecd79b458c" gracePeriod=25
Nov 02 12:33:15.381651 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:15.381584    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2" podUID="42089fd9-dfb3-4681-8a92-c7b66ee2b7fb" containerName="payload-pre-processing" containerID="cri-o://f5bad896d78525469477c73e3b7fcf7258160a5b69e9f3ce6bd84f29c456b6fa" gracePeriod=25
Nov 02 12:33:16.308829 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.308786    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv" podUID="d829dab9-d274-485e-bef3-a29c5ddfff22" containerName="main" containerID="cri-o://5c2fc959481138c73b34d36eabd1e1830ab8bd27770e1c87f0448a1e8ff15b5a" gracePeriod=45
Nov 02 12:33:16.576123 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.576060    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:33:16.677706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.677677    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-model-cache\") pod \"d829dab9-d274-485e-bef3-a29c5ddfff22\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") "
Nov 02 12:33:16.677916 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.677732    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-dshm\") pod \"d829dab9-d274-485e-bef3-a29c5ddfff22\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") "
Nov 02 12:33:16.677916 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.677757    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d829dab9-d274-485e-bef3-a29c5ddfff22-tls-certs\" (UniqueName: \"kubernetes.io/secret/d829dab9-d274-485e-bef3-a29c5ddfff22-tls-certs\") pod \"d829dab9-d274-485e-bef3-a29c5ddfff22\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") "
Nov 02 12:33:16.677916 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.677812    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-tmp-dir\") pod \"d829dab9-d274-485e-bef3-a29c5ddfff22\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") "
Nov 02 12:33:16.677916 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.677836    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-home\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-home\") pod \"d829dab9-d274-485e-bef3-a29c5ddfff22\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") "
Nov 02 12:33:16.678164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.677931    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d829dab9-d274-485e-bef3-a29c5ddfff22-kube-api-access-7rzp6\" (UniqueName: \"kubernetes.io/projected/d829dab9-d274-485e-bef3-a29c5ddfff22-kube-api-access-7rzp6\") pod \"d829dab9-d274-485e-bef3-a29c5ddfff22\" (UID: \"d829dab9-d274-485e-bef3-a29c5ddfff22\") "
Nov 02 12:33:16.678164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.677990    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-model-cache" pod "d829dab9-d274-485e-bef3-a29c5ddfff22" (UID: "d829dab9-d274-485e-bef3-a29c5ddfff22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:33:16.678164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.678109    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-tmp-dir" pod "d829dab9-d274-485e-bef3-a29c5ddfff22" (UID: "d829dab9-d274-485e-bef3-a29c5ddfff22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:33:16.678164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.678143    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-home" pod "d829dab9-d274-485e-bef3-a29c5ddfff22" (UID: "d829dab9-d274-485e-bef3-a29c5ddfff22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:33:16.678431 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.678410    4464 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-model-cache\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:16.678431 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.678431    4464 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-tmp-dir\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:16.678642 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.678442    4464 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-home\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:16.680128 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.680105    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d829dab9-d274-485e-bef3-a29c5ddfff22-kube-api-access-7rzp6" pod "d829dab9-d274-485e-bef3-a29c5ddfff22" (UID: "d829dab9-d274-485e-bef3-a29c5ddfff22"). InnerVolumeSpecName "kube-api-access-7rzp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:33:16.680221 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.680109    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d829dab9-d274-485e-bef3-a29c5ddfff22-tls-certs" pod "d829dab9-d274-485e-bef3-a29c5ddfff22" (UID: "d829dab9-d274-485e-bef3-a29c5ddfff22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:33:16.680221 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.680107    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-dshm" pod "d829dab9-d274-485e-bef3-a29c5ddfff22" (UID: "d829dab9-d274-485e-bef3-a29c5ddfff22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:33:16.780309 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.780266    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rzp6\" (UniqueName: \"kubernetes.io/projected/d829dab9-d274-485e-bef3-a29c5ddfff22-kube-api-access-7rzp6\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:16.780309 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.780307    4464 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d829dab9-d274-485e-bef3-a29c5ddfff22-dshm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:16.780504 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:16.780322    4464 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d829dab9-d274-485e-bef3-a29c5ddfff22-tls-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:17.258593 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:17.258555    4464 generic.go:356] "Generic (PLEG): container finished" podID="d829dab9-d274-485e-bef3-a29c5ddfff22" containerID="5c2fc959481138c73b34d36eabd1e1830ab8bd27770e1c87f0448a1e8ff15b5a" exitCode=0
Nov 02 12:33:17.258796 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:17.258626    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"
Nov 02 12:33:17.258796 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:17.258632    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv" event={"ID":"d829dab9-d274-485e-bef3-a29c5ddfff22","Type":"ContainerDied","Data":"5c2fc959481138c73b34d36eabd1e1830ab8bd27770e1c87f0448a1e8ff15b5a"}
Nov 02 12:33:17.258796 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:17.258675    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv" event={"ID":"d829dab9-d274-485e-bef3-a29c5ddfff22","Type":"ContainerDied","Data":"90ce8ede3f467618e179bf1507ac057f952a2ca9c7a2d803ca9c1507ae137af1"}
Nov 02 12:33:17.258796 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:17.258693    4464 scope.go:122] "RemoveContainer" containerID="5c2fc959481138c73b34d36eabd1e1830ab8bd27770e1c87f0448a1e8ff15b5a"
Nov 02 12:33:17.268906 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:17.268889    4464 scope.go:122] "RemoveContainer" containerID="5c2fc959481138c73b34d36eabd1e1830ab8bd27770e1c87f0448a1e8ff15b5a"
Nov 02 12:33:17.269200 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:33:17.269182    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2fc959481138c73b34d36eabd1e1830ab8bd27770e1c87f0448a1e8ff15b5a\": container with ID starting with 5c2fc959481138c73b34d36eabd1e1830ab8bd27770e1c87f0448a1e8ff15b5a not found: ID does not exist" containerID="5c2fc959481138c73b34d36eabd1e1830ab8bd27770e1c87f0448a1e8ff15b5a"
Nov 02 12:33:17.269248 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:17.269209    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2fc959481138c73b34d36eabd1e1830ab8bd27770e1c87f0448a1e8ff15b5a"} err="failed to get container status \"5c2fc959481138c73b34d36eabd1e1830ab8bd27770e1c87f0448a1e8ff15b5a\": rpc error: code = NotFound desc = could not find container \"5c2fc959481138c73b34d36eabd1e1830ab8bd27770e1c87f0448a1e8ff15b5a\": container with ID starting with 5c2fc959481138c73b34d36eabd1e1830ab8bd27770e1c87f0448a1e8ff15b5a not found: ID does not exist"
Nov 02 12:33:17.425892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:17.425831    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"]
Nov 02 12:33:17.476468 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:17.476436    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/distinct-2-w3-4b9210-kserve-768464c84f-pg4mv"]
Nov 02 12:33:17.583301 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:17.583231    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d829dab9-d274-485e-bef3-a29c5ddfff22" path="/var/lib/kubelet/pods/d829dab9-d274-485e-bef3-a29c5ddfff22/volumes"
Nov 02 12:33:21.829527 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:21.829493    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2" podUID="42089fd9-dfb3-4681-8a92-c7b66ee2b7fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:33:21.829939 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:21.829567    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr" podUID="1b98a955-089d-4f50-a719-4918f7f5a2c0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:33:23.009739 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:33:23.009702    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ab1ff_866c_4d73_bccb_7eded744c791.slice/crio-conmon-5587c946fad483f440f7f4ccc94254ed2a658671f453c373d06e24b90fbd7b7c.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:33:24.810992 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:33:24.810941    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ab1ff_866c_4d73_bccb_7eded744c791.slice/crio-conmon-5587c946fad483f440f7f4ccc94254ed2a658671f453c373d06e24b90fbd7b7c.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:33:25.896846 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:33:25.896804    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ab1ff_866c_4d73_bccb_7eded744c791.slice/crio-conmon-5587c946fad483f440f7f4ccc94254ed2a658671f453c373d06e24b90fbd7b7c.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:33:25.897355 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:33:25.896858    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ab1ff_866c_4d73_bccb_7eded744c791.slice/crio-conmon-5587c946fad483f440f7f4ccc94254ed2a658671f453c373d06e24b90fbd7b7c.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:33:28.292493 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:33:28.292457    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:33:31.829498 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:31.829460    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr" podUID="1b98a955-089d-4f50-a719-4918f7f5a2c0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:33:31.829958 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:31.829512    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2" podUID="42089fd9-dfb3-4681-8a92-c7b66ee2b7fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:33:34.875922 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:33:34.875888    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ab1ff_866c_4d73_bccb_7eded744c791.slice/crio-conmon-5587c946fad483f440f7f4ccc94254ed2a658671f453c373d06e24b90fbd7b7c.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:33:35.350816 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:35.350784    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:33:35.431200 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:35.431161    4464 generic.go:356] "Generic (PLEG): container finished" podID="1b98a955-089d-4f50-a719-4918f7f5a2c0" containerID="3da45036038f9218a498bce32525296e6c6966b93c83900dcbf0cfecd79b458c" exitCode=0
Nov 02 12:33:35.431352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:35.431225    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr" event={"ID":"1b98a955-089d-4f50-a719-4918f7f5a2c0","Type":"ContainerDied","Data":"3da45036038f9218a498bce32525296e6c6966b93c83900dcbf0cfecd79b458c"}
Nov 02 12:33:35.564663 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:35.564633    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"
Nov 02 12:33:35.665026 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:35.660595    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b98a955-089d-4f50-a719-4918f7f5a2c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98a955-089d-4f50-a719-4918f7f5a2c0-plugins-config-volume\") pod \"1b98a955-089d-4f50-a719-4918f7f5a2c0\" (UID: \"1b98a955-089d-4f50-a719-4918f7f5a2c0\") "
Nov 02 12:33:35.665026 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:35.660803    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b98a955-089d-4f50-a719-4918f7f5a2c0-kube-api-access-974fz\" (UniqueName: \"kubernetes.io/projected/1b98a955-089d-4f50-a719-4918f7f5a2c0-kube-api-access-974fz\") pod \"1b98a955-089d-4f50-a719-4918f7f5a2c0\" (UID: \"1b98a955-089d-4f50-a719-4918f7f5a2c0\") "
Nov 02 12:33:35.669098 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:35.669053    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b98a955-089d-4f50-a719-4918f7f5a2c0-plugins-config-volume" pod "1b98a955-089d-4f50-a719-4918f7f5a2c0" (UID: "1b98a955-089d-4f50-a719-4918f7f5a2c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:33:35.717435 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:35.717392    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b98a955-089d-4f50-a719-4918f7f5a2c0-kube-api-access-974fz" pod "1b98a955-089d-4f50-a719-4918f7f5a2c0" (UID: "1b98a955-089d-4f50-a719-4918f7f5a2c0"). InnerVolumeSpecName "kube-api-access-974fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:33:35.734454 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:35.734425    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"
Nov 02 12:33:35.762943 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:35.762916    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42089fd9-dfb3-4681-8a92-c7b66ee2b7fb-kube-api-access-8w8zp\" (UniqueName: \"kubernetes.io/projected/42089fd9-dfb3-4681-8a92-c7b66ee2b7fb-kube-api-access-8w8zp\") pod \"42089fd9-dfb3-4681-8a92-c7b66ee2b7fb\" (UID: \"42089fd9-dfb3-4681-8a92-c7b66ee2b7fb\") "
Nov 02 12:33:35.763124 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:35.762952    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42089fd9-dfb3-4681-8a92-c7b66ee2b7fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42089fd9-dfb3-4681-8a92-c7b66ee2b7fb-plugins-config-volume\") pod \"42089fd9-dfb3-4681-8a92-c7b66ee2b7fb\" (UID: \"42089fd9-dfb3-4681-8a92-c7b66ee2b7fb\") "
Nov 02 12:33:35.763408 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:35.763379    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42089fd9-dfb3-4681-8a92-c7b66ee2b7fb-plugins-config-volume" pod "42089fd9-dfb3-4681-8a92-c7b66ee2b7fb" (UID: "42089fd9-dfb3-4681-8a92-c7b66ee2b7fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:33:35.763476 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:35.763458    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-974fz\" (UniqueName: \"kubernetes.io/projected/1b98a955-089d-4f50-a719-4918f7f5a2c0-kube-api-access-974fz\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:35.763513 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:35.763484    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b98a955-089d-4f50-a719-4918f7f5a2c0-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:35.765318 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:35.765294    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42089fd9-dfb3-4681-8a92-c7b66ee2b7fb-kube-api-access-8w8zp" pod "42089fd9-dfb3-4681-8a92-c7b66ee2b7fb" (UID: "42089fd9-dfb3-4681-8a92-c7b66ee2b7fb"). InnerVolumeSpecName "kube-api-access-8w8zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:33:35.865221 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:35.865134    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w8zp\" (UniqueName: \"kubernetes.io/projected/42089fd9-dfb3-4681-8a92-c7b66ee2b7fb-kube-api-access-8w8zp\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:35.865221 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:35.865168    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42089fd9-dfb3-4681-8a92-c7b66ee2b7fb-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:36.442997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:36.441954    4464 generic.go:356] "Generic (PLEG): container finished" podID="42089fd9-dfb3-4681-8a92-c7b66ee2b7fb" containerID="f5bad896d78525469477c73e3b7fcf7258160a5b69e9f3ce6bd84f29c456b6fa" exitCode=0
Nov 02 12:33:36.442997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:36.442028    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"
Nov 02 12:33:36.442997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:36.442032    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2" event={"ID":"42089fd9-dfb3-4681-8a92-c7b66ee2b7fb","Type":"ContainerDied","Data":"f5bad896d78525469477c73e3b7fcf7258160a5b69e9f3ce6bd84f29c456b6fa"}
Nov 02 12:33:36.442997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:36.442110    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2" event={"ID":"42089fd9-dfb3-4681-8a92-c7b66ee2b7fb","Type":"ContainerDied","Data":"5104ef8e61fec056a5f5507831a7218e97a88f28d02ab5518ea76c9e141bf705"}
Nov 02 12:33:36.442997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:36.442127    4464 scope.go:122] "RemoveContainer" containerID="f5bad896d78525469477c73e3b7fcf7258160a5b69e9f3ce6bd84f29c456b6fa"
Nov 02 12:33:36.444267 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:36.444238    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr" event={"ID":"1b98a955-089d-4f50-a719-4918f7f5a2c0","Type":"ContainerDied","Data":"2e1496526234d589b1ceca239c7823b1ba564a06c7496c2a295211512145764b"}
Nov 02 12:33:36.444420 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:36.444270    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"
Nov 02 12:33:36.457446 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:36.457422    4464 scope.go:122] "RemoveContainer" containerID="f5bad896d78525469477c73e3b7fcf7258160a5b69e9f3ce6bd84f29c456b6fa"
Nov 02 12:33:36.457727 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:33:36.457708    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5bad896d78525469477c73e3b7fcf7258160a5b69e9f3ce6bd84f29c456b6fa\": container with ID starting with f5bad896d78525469477c73e3b7fcf7258160a5b69e9f3ce6bd84f29c456b6fa not found: ID does not exist" containerID="f5bad896d78525469477c73e3b7fcf7258160a5b69e9f3ce6bd84f29c456b6fa"
Nov 02 12:33:36.457784 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:36.457734    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5bad896d78525469477c73e3b7fcf7258160a5b69e9f3ce6bd84f29c456b6fa"} err="failed to get container status \"f5bad896d78525469477c73e3b7fcf7258160a5b69e9f3ce6bd84f29c456b6fa\": rpc error: code = NotFound desc = could not find container \"f5bad896d78525469477c73e3b7fcf7258160a5b69e9f3ce6bd84f29c456b6fa\": container with ID starting with f5bad896d78525469477c73e3b7fcf7258160a5b69e9f3ce6bd84f29c456b6fa not found: ID does not exist"
Nov 02 12:33:36.457784 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:36.457751    4464 scope.go:122] "RemoveContainer" containerID="3da45036038f9218a498bce32525296e6c6966b93c83900dcbf0cfecd79b458c"
Nov 02 12:33:36.615007 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:36.614650    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"]
Nov 02 12:33:36.619609 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:36.619580    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4b9210-bb856757c-b8bnr"]
Nov 02 12:33:36.726790 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:36.726760    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"]
Nov 02 12:33:36.729543 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:36.729522    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4b9210-79ddd5fbc-k58k2"]
Nov 02 12:33:37.582484 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:37.582432    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b98a955-089d-4f50-a719-4918f7f5a2c0" path="/var/lib/kubelet/pods/1b98a955-089d-4f50-a719-4918f7f5a2c0/volumes"
Nov 02 12:33:37.583043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:37.583003    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42089fd9-dfb3-4681-8a92-c7b66ee2b7fb" path="/var/lib/kubelet/pods/42089fd9-dfb3-4681-8a92-c7b66ee2b7fb/volumes"
Nov 02 12:33:38.032043 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:33:38.031970    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ab1ff_866c_4d73_bccb_7eded744c791.slice/crio-conmon-5587c946fad483f440f7f4ccc94254ed2a658671f453c373d06e24b90fbd7b7c.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:33:44.656583 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:44.656546    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"]
Nov 02 12:33:44.940929 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:33:44.940838    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ab1ff_866c_4d73_bccb_7eded744c791.slice/crio-conmon-5587c946fad483f440f7f4ccc94254ed2a658671f453c373d06e24b90fbd7b7c.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:33:49.656960 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:49.656919    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv" podUID="a3d1d64c-7010-410b-ac86-54b9d481e8ac" containerName="payload-processing" containerID="cri-o://3ad150c0ec7297fc455491e17d1193140b702cdd20d9265702d97a8085cbea76" gracePeriod=25
Nov 02 12:33:49.922678 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:49.922610    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"
Nov 02 12:33:49.971425 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:49.971398    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3d1d64c-7010-410b-ac86-54b9d481e8ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d1d64c-7010-410b-ac86-54b9d481e8ac-plugins-config-volume\") pod \"a3d1d64c-7010-410b-ac86-54b9d481e8ac\" (UID: \"a3d1d64c-7010-410b-ac86-54b9d481e8ac\") "
Nov 02 12:33:49.971546 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:49.971432    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d1d64c-7010-410b-ac86-54b9d481e8ac-kube-api-access-htrsz\" (UniqueName: \"kubernetes.io/projected/a3d1d64c-7010-410b-ac86-54b9d481e8ac-kube-api-access-htrsz\") pod \"a3d1d64c-7010-410b-ac86-54b9d481e8ac\" (UID: \"a3d1d64c-7010-410b-ac86-54b9d481e8ac\") "
Nov 02 12:33:49.971734 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:49.971713    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d1d64c-7010-410b-ac86-54b9d481e8ac-plugins-config-volume" pod "a3d1d64c-7010-410b-ac86-54b9d481e8ac" (UID: "a3d1d64c-7010-410b-ac86-54b9d481e8ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:33:49.971870 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:49.971848    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3d1d64c-7010-410b-ac86-54b9d481e8ac-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:49.973415 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:49.973394    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d1d64c-7010-410b-ac86-54b9d481e8ac-kube-api-access-htrsz" pod "a3d1d64c-7010-410b-ac86-54b9d481e8ac" (UID: "a3d1d64c-7010-410b-ac86-54b9d481e8ac"). InnerVolumeSpecName "kube-api-access-htrsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:33:50.073025 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:50.072965    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htrsz\" (UniqueName: \"kubernetes.io/projected/a3d1d64c-7010-410b-ac86-54b9d481e8ac-kube-api-access-htrsz\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:33:50.208603 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:50.208522    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:33:50.616001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:50.608245    4464 generic.go:356] "Generic (PLEG): container finished" podID="a3d1d64c-7010-410b-ac86-54b9d481e8ac" containerID="3ad150c0ec7297fc455491e17d1193140b702cdd20d9265702d97a8085cbea76" exitCode=0
Nov 02 12:33:50.616001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:50.608461    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv" event={"ID":"a3d1d64c-7010-410b-ac86-54b9d481e8ac","Type":"ContainerDied","Data":"3ad150c0ec7297fc455491e17d1193140b702cdd20d9265702d97a8085cbea76"}
Nov 02 12:33:50.616001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:50.608525    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv" event={"ID":"a3d1d64c-7010-410b-ac86-54b9d481e8ac","Type":"ContainerDied","Data":"28bcf2085278e327e3d337b165c4baacb648370da013b7302f936510ee576987"}
Nov 02 12:33:50.616001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:50.608557    4464 scope.go:122] "RemoveContainer" containerID="3ad150c0ec7297fc455491e17d1193140b702cdd20d9265702d97a8085cbea76"
Nov 02 12:33:50.616001 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:50.608754    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"
Nov 02 12:33:50.692727 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:50.692681    4464 scope.go:122] "RemoveContainer" containerID="3ad150c0ec7297fc455491e17d1193140b702cdd20d9265702d97a8085cbea76"
Nov 02 12:33:50.693461 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:33:50.693425    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad150c0ec7297fc455491e17d1193140b702cdd20d9265702d97a8085cbea76\": container with ID starting with 3ad150c0ec7297fc455491e17d1193140b702cdd20d9265702d97a8085cbea76 not found: ID does not exist" containerID="3ad150c0ec7297fc455491e17d1193140b702cdd20d9265702d97a8085cbea76"
Nov 02 12:33:50.693674 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:50.693638    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad150c0ec7297fc455491e17d1193140b702cdd20d9265702d97a8085cbea76"} err="failed to get container status \"3ad150c0ec7297fc455491e17d1193140b702cdd20d9265702d97a8085cbea76\": rpc error: code = NotFound desc = could not find container \"3ad150c0ec7297fc455491e17d1193140b702cdd20d9265702d97a8085cbea76\": container with ID starting with 3ad150c0ec7297fc455491e17d1193140b702cdd20d9265702d97a8085cbea76 not found: ID does not exist"
Nov 02 12:33:50.782769 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:50.782739    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"]
Nov 02 12:33:50.833783 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:50.833752    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-cf0095-5dd5579d5f-gg9mv"]
Nov 02 12:33:51.582165 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.582128    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3d1d64c-7010-410b-ac86-54b9d481e8ac" path="/var/lib/kubelet/pods/a3d1d64c-7010-410b-ac86-54b9d481e8ac/volumes"
Nov 02 12:33:51.871572 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.871483    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"]
Nov 02 12:33:51.873178 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.873148    4464 cpu_manager.go:414] "containerMap: removing container" podUID="a3d1d64c-7010-410b-ac86-54b9d481e8ac" containerName="payload-processing"
Nov 02 12:33:51.873178 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.873178    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d1d64c-7010-410b-ac86-54b9d481e8ac" containerName="payload-processing"
Nov 02 12:33:51.873389 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.873208    4464 cpu_manager.go:414] "containerMap: removing container" podUID="d829dab9-d274-485e-bef3-a29c5ddfff22" containerName="main"
Nov 02 12:33:51.873389 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.873215    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d829dab9-d274-485e-bef3-a29c5ddfff22" containerName="main"
Nov 02 12:33:51.873389 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.873227    4464 cpu_manager.go:414] "containerMap: removing container" podUID="42089fd9-dfb3-4681-8a92-c7b66ee2b7fb" containerName="payload-pre-processing"
Nov 02 12:33:51.873389 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.873234    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42089fd9-dfb3-4681-8a92-c7b66ee2b7fb" containerName="payload-pre-processing"
Nov 02 12:33:51.873389 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.873257    4464 cpu_manager.go:414] "containerMap: removing container" podUID="bb7ab1ff-866c-4d73-bccb-7eded744c791" containerName="revoke-keys"
Nov 02 12:33:51.873389 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.873265    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7ab1ff-866c-4d73-bccb-7eded744c791" containerName="revoke-keys"
Nov 02 12:33:51.873389 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.873305    4464 cpu_manager.go:414] "containerMap: removing container" podUID="1b98a955-089d-4f50-a719-4918f7f5a2c0" containerName="payload-processing"
Nov 02 12:33:51.873389 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.873311    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b98a955-089d-4f50-a719-4918f7f5a2c0" containerName="payload-processing"
Nov 02 12:33:51.873801 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.873429    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b98a955-089d-4f50-a719-4918f7f5a2c0" containerName="payload-processing"
Nov 02 12:33:51.873801 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.873446    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d1d64c-7010-410b-ac86-54b9d481e8ac" containerName="payload-processing"
Nov 02 12:33:51.873801 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.873455    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="d829dab9-d274-485e-bef3-a29c5ddfff22" containerName="main"
Nov 02 12:33:51.873801 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.873463    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="42089fd9-dfb3-4681-8a92-c7b66ee2b7fb" containerName="payload-pre-processing"
Nov 02 12:33:51.873801 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.873471    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7ab1ff-866c-4d73-bccb-7eded744c791" containerName="revoke-keys"
Nov 02 12:33:51.905038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.898589    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:51.911957 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:51.909447    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"]
Nov 02 12:33:52.005753 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.005725    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.005932 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.005785    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.005932 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.005874    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.005932 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.005924    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.006149 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.006024    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.006149 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.006059    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3befcd84-95ad-4eed-88ad-eb6d332a2048-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.006225 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.006167    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.006225 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.006195    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.006355 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.006337    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chk5f\" (UniqueName: \"kubernetes.io/projected/3befcd84-95ad-4eed-88ad-eb6d332a2048-kube-api-access-chk5f\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.107172 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.107146    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.107350 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.107179    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3befcd84-95ad-4eed-88ad-eb6d332a2048-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.107350 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.107230    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.107350 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.107274    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.107588 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.107391    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chk5f\" (UniqueName: \"kubernetes.io/projected/3befcd84-95ad-4eed-88ad-eb6d332a2048-kube-api-access-chk5f\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.107588 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.107464    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.107588 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.107545    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.107588 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.107558    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.107804 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.107590    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.107804 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.107625    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.107926 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.107800    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.107926 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.107912    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.108070 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.108021    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.108685 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.108662    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3befcd84-95ad-4eed-88ad-eb6d332a2048-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.109652 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.109630    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.110173 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.110155    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.169467 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.169399    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.169575 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.169518    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chk5f\" (UniqueName: \"kubernetes.io/projected/3befcd84-95ad-4eed-88ad-eb6d332a2048-kube-api-access-chk5f\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.226102 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.226071    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:52.657106 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.657077    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"]
Nov 02 12:33:52.657254 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:33:52.657232    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3befcd84_95ad_4eed_88ad_eb6d332a2048.slice/crio-bae601c093082ba41d9409948078f506586cc5fd9a12a0bd3580913305936041 WatchSource:0}: Error finding container bae601c093082ba41d9409948078f506586cc5fd9a12a0bd3580913305936041: Status 404 returned error can't find the container with id bae601c093082ba41d9409948078f506586cc5fd9a12a0bd3580913305936041
Nov 02 12:33:52.659523 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.659491    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:33:52.659618 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.659561    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:33:52.659618 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:52.659593    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:33:52.981200 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:33:52.981168    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ab1ff_866c_4d73_bccb_7eded744c791.slice/crio-conmon-5587c946fad483f440f7f4ccc94254ed2a658671f453c373d06e24b90fbd7b7c.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:33:53.632023 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:53.631955    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5" event={"ID":"3befcd84-95ad-4eed-88ad-eb6d332a2048","Type":"ContainerStarted","Data":"835687479f2f37b1649cd0ff745e0131eb0c12fe278ba5ed5f37d5abcc10e558"}
Nov 02 12:33:53.632204 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:53.632052    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5" event={"ID":"3befcd84-95ad-4eed-88ad-eb6d332a2048","Type":"ContainerStarted","Data":"bae601c093082ba41d9409948078f506586cc5fd9a12a0bd3580913305936041"}
Nov 02 12:33:53.856474 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:53.856423    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5" podStartSLOduration=2.856409438 podStartE2EDuration="2.856409438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:33:53.805206052 +0000 UTC m=+2262.368914463" watchObservedRunningTime="2026-11-02 12:33:53.856409438 +0000 UTC m=+2262.420117847"
Nov 02 12:33:54.227292 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:54.227245    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:33:54.228780 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:54.228758    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 02 12:33:54.228844 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:54.228803    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5" podUID="3befcd84-95ad-4eed-88ad-eb6d332a2048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 02 12:33:55.026061 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:33:55.026027    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ab1ff_866c_4d73_bccb_7eded744c791.slice/crio-conmon-5587c946fad483f440f7f4ccc94254ed2a658671f453c373d06e24b90fbd7b7c.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:33:55.226567 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:55.226531    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 02 12:33:55.226730 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:55.226589    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5" podUID="3befcd84-95ad-4eed-88ad-eb6d332a2048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 02 12:33:56.227217 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:56.227183    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 02 12:33:56.227594 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:56.227250    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5" podUID="3befcd84-95ad-4eed-88ad-eb6d332a2048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 02 12:33:57.227258 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:57.227223    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 02 12:33:57.227645 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:57.227278    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5" podUID="3befcd84-95ad-4eed-88ad-eb6d332a2048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 02 12:33:58.226857 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:58.226818    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 02 12:33:58.227068 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:58.226873    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5" podUID="3befcd84-95ad-4eed-88ad-eb6d332a2048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 02 12:33:58.311568 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:33:58.311521    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:33:59.169724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:59.169690    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"]
Nov 02 12:33:59.169925 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:59.169884    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerName="istio-proxy" containerID="cri-o://1d395adc2e1c2293dd782593366d6e57e6dec93917a756c5734c605d732501e1" gracePeriod=30
Nov 02 12:33:59.180854 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:59.180828    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"]
Nov 02 12:33:59.181043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:33:59.181024    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5" podUID="3befcd84-95ad-4eed-88ad-eb6d332a2048" containerName="istio-proxy" containerID="cri-o://835687479f2f37b1649cd0ff745e0131eb0c12fe278ba5ed5f37d5abcc10e558" gracePeriod=30
Nov 02 12:34:02.419538 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:02.419507    4464 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:34:02.419915 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:02.419558    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:34:04.442548 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.442478    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:34:04.478999 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.478964    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:34:04.540761 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.540736    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-credential-socket\") pod \"3befcd84-95ad-4eed-88ad-eb6d332a2048\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") "
Nov 02 12:34:04.540907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.540775    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-token\" (UniqueName: \"kubernetes.io/projected/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-token\") pod \"3befcd84-95ad-4eed-88ad-eb6d332a2048\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") "
Nov 02 12:34:04.540907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.540812    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-token\" (UniqueName: \"kubernetes.io/projected/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-token\") pod \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") "
Nov 02 12:34:04.540907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.540871    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-data\") pod \"3befcd84-95ad-4eed-88ad-eb6d332a2048\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") "
Nov 02 12:34:04.540907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.540900    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-envoy\") pod \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") "
Nov 02 12:34:04.541181 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.540939    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-credential-socket\") pod \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") "
Nov 02 12:34:04.541181 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.540966    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istiod-ca-cert\") pod \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") "
Nov 02 12:34:04.541181 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541054    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-workload-certs\") pod \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") "
Nov 02 12:34:04.541181 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541069    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-credential-socket" pod "3befcd84-95ad-4eed-88ad-eb6d332a2048" (UID: "3befcd84-95ad-4eed-88ad-eb6d332a2048"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:04.541181 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541113    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-kube-api-access-vgkk5\" (UniqueName: \"kubernetes.io/projected/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-kube-api-access-vgkk5\") pod \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") "
Nov 02 12:34:04.541181 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541142    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3befcd84-95ad-4eed-88ad-eb6d332a2048-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3befcd84-95ad-4eed-88ad-eb6d332a2048-istiod-ca-cert\") pod \"3befcd84-95ad-4eed-88ad-eb6d332a2048\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") "
Nov 02 12:34:04.541505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541187    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-workload-socket\") pod \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") "
Nov 02 12:34:04.541505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541219    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3befcd84-95ad-4eed-88ad-eb6d332a2048-kube-api-access-chk5f\" (UniqueName: \"kubernetes.io/projected/3befcd84-95ad-4eed-88ad-eb6d332a2048-kube-api-access-chk5f\") pod \"3befcd84-95ad-4eed-88ad-eb6d332a2048\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") "
Nov 02 12:34:04.541505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541257    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-credential-socket" pod "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" (UID: "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:04.541505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541317    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-podinfo\") pod \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") "
Nov 02 12:34:04.541505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541363    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-podinfo\") pod \"3befcd84-95ad-4eed-88ad-eb6d332a2048\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") "
Nov 02 12:34:04.541505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541377    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-workload-certs" pod "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" (UID: "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:04.541505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541388    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-data" pod "3befcd84-95ad-4eed-88ad-eb6d332a2048" (UID: "3befcd84-95ad-4eed-88ad-eb6d332a2048"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:04.541505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541442    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-data\") pod \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\" (UID: \"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7\") "
Nov 02 12:34:04.541505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541470    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-workload-socket\") pod \"3befcd84-95ad-4eed-88ad-eb6d332a2048\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") "
Nov 02 12:34:04.542037 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541516    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-workload-certs\") pod \"3befcd84-95ad-4eed-88ad-eb6d332a2048\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") "
Nov 02 12:34:04.542037 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541531    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istiod-ca-cert" pod "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" (UID: "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:34:04.542037 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541570    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-envoy\") pod \"3befcd84-95ad-4eed-88ad-eb6d332a2048\" (UID: \"3befcd84-95ad-4eed-88ad-eb6d332a2048\") "
Nov 02 12:34:04.542037 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541598    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3befcd84-95ad-4eed-88ad-eb6d332a2048-istiod-ca-cert" pod "3befcd84-95ad-4eed-88ad-eb6d332a2048" (UID: "3befcd84-95ad-4eed-88ad-eb6d332a2048"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:34:04.542037 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.541836    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-workload-socket" pod "3befcd84-95ad-4eed-88ad-eb6d332a2048" (UID: "3befcd84-95ad-4eed-88ad-eb6d332a2048"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:04.542302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.542039    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-data" pod "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" (UID: "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:04.542302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.542099    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.542302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.542119    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.542302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.542136    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.542302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.542151    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.542302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.542159    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-workload-certs" pod "3befcd84-95ad-4eed-88ad-eb6d332a2048" (UID: "3befcd84-95ad-4eed-88ad-eb6d332a2048"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:04.542302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.542166    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.542302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.542209    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.542302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.542224    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3befcd84-95ad-4eed-88ad-eb6d332a2048-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.542302 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.542244    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-workload-socket" pod "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" (UID: "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:04.544801 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.544356    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-envoy" pod "3befcd84-95ad-4eed-88ad-eb6d332a2048" (UID: "3befcd84-95ad-4eed-88ad-eb6d332a2048"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:04.544801 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.544566    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-token" pod "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" (UID: "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:34:04.545042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.544907    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-kube-api-access-vgkk5" pod "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" (UID: "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7"). InnerVolumeSpecName "kube-api-access-vgkk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:34:04.545042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.545019    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3befcd84-95ad-4eed-88ad-eb6d332a2048-kube-api-access-chk5f" pod "3befcd84-95ad-4eed-88ad-eb6d332a2048" (UID: "3befcd84-95ad-4eed-88ad-eb6d332a2048"). InnerVolumeSpecName "kube-api-access-chk5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:34:04.545185 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.545113    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-podinfo" pod "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" (UID: "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:34:04.545263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.545240    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-envoy" pod "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" (UID: "5d29abcb-5367-49e7-b4b0-4c27e87fe2a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:04.545316 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.545239    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-podinfo" pod "3befcd84-95ad-4eed-88ad-eb6d332a2048" (UID: "3befcd84-95ad-4eed-88ad-eb6d332a2048"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:34:04.545316 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.545273    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-token" pod "3befcd84-95ad-4eed-88ad-eb6d332a2048" (UID: "3befcd84-95ad-4eed-88ad-eb6d332a2048"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:34:04.643884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.643844    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.643884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.643878    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.644131 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.643893    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.644131 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.643907    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.644131 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.643921    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.644131 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.643936    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.644131 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.643950    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgkk5\" (UniqueName: \"kubernetes.io/projected/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-kube-api-access-vgkk5\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.644131 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.643965    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.644131 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.643994    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chk5f\" (UniqueName: \"kubernetes.io/projected/3befcd84-95ad-4eed-88ad-eb6d332a2048-kube-api-access-chk5f\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.644131 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.644007    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.644131 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.644021    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3befcd84-95ad-4eed-88ad-eb6d332a2048-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:04.735468 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.735399    4464 generic.go:356] "Generic (PLEG): container finished" podID="3befcd84-95ad-4eed-88ad-eb6d332a2048" containerID="835687479f2f37b1649cd0ff745e0131eb0c12fe278ba5ed5f37d5abcc10e558" exitCode=0
Nov 02 12:34:04.735829 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.735474    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"
Nov 02 12:34:04.735829 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.735500    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5" event={"ID":"3befcd84-95ad-4eed-88ad-eb6d332a2048","Type":"ContainerDied","Data":"835687479f2f37b1649cd0ff745e0131eb0c12fe278ba5ed5f37d5abcc10e558"}
Nov 02 12:34:04.735829 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.735596    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5" event={"ID":"3befcd84-95ad-4eed-88ad-eb6d332a2048","Type":"ContainerDied","Data":"bae601c093082ba41d9409948078f506586cc5fd9a12a0bd3580913305936041"}
Nov 02 12:34:04.735829 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.735626    4464 scope.go:122] "RemoveContainer" containerID="835687479f2f37b1649cd0ff745e0131eb0c12fe278ba5ed5f37d5abcc10e558"
Nov 02 12:34:04.740407 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.738483    4464 generic.go:356] "Generic (PLEG): container finished" podID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerID="1d395adc2e1c2293dd782593366d6e57e6dec93917a756c5734c605d732501e1" exitCode=0
Nov 02 12:34:04.740407 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.738646    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" event={"ID":"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7","Type":"ContainerDied","Data":"1d395adc2e1c2293dd782593366d6e57e6dec93917a756c5734c605d732501e1"}
Nov 02 12:34:04.740407 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.738692    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87" event={"ID":"5d29abcb-5367-49e7-b4b0-4c27e87fe2a7","Type":"ContainerDied","Data":"7b24aec605dfcb01a3bb052fc89a77a858a6f816772da962268914346e087794"}
Nov 02 12:34:04.740407 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.738773    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"
Nov 02 12:34:04.748892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.748875    4464 scope.go:122] "RemoveContainer" containerID="835687479f2f37b1649cd0ff745e0131eb0c12fe278ba5ed5f37d5abcc10e558"
Nov 02 12:34:04.749223 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:34:04.749204    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835687479f2f37b1649cd0ff745e0131eb0c12fe278ba5ed5f37d5abcc10e558\": container with ID starting with 835687479f2f37b1649cd0ff745e0131eb0c12fe278ba5ed5f37d5abcc10e558 not found: ID does not exist" containerID="835687479f2f37b1649cd0ff745e0131eb0c12fe278ba5ed5f37d5abcc10e558"
Nov 02 12:34:04.749321 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.749233    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835687479f2f37b1649cd0ff745e0131eb0c12fe278ba5ed5f37d5abcc10e558"} err="failed to get container status \"835687479f2f37b1649cd0ff745e0131eb0c12fe278ba5ed5f37d5abcc10e558\": rpc error: code = NotFound desc = could not find container \"835687479f2f37b1649cd0ff745e0131eb0c12fe278ba5ed5f37d5abcc10e558\": container with ID starting with 835687479f2f37b1649cd0ff745e0131eb0c12fe278ba5ed5f37d5abcc10e558 not found: ID does not exist"
Nov 02 12:34:04.749321 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.749258    4464 scope.go:122] "RemoveContainer" containerID="1d395adc2e1c2293dd782593366d6e57e6dec93917a756c5734c605d732501e1"
Nov 02 12:34:04.762053 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.762035    4464 scope.go:122] "RemoveContainer" containerID="886d0c3fe1f711852cb41aa5416170ccc082175ccb5d914f176925e7349f7fcd"
Nov 02 12:34:04.773268 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.773250    4464 scope.go:122] "RemoveContainer" containerID="1d395adc2e1c2293dd782593366d6e57e6dec93917a756c5734c605d732501e1"
Nov 02 12:34:04.773521 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:34:04.773504    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d395adc2e1c2293dd782593366d6e57e6dec93917a756c5734c605d732501e1\": container with ID starting with 1d395adc2e1c2293dd782593366d6e57e6dec93917a756c5734c605d732501e1 not found: ID does not exist" containerID="1d395adc2e1c2293dd782593366d6e57e6dec93917a756c5734c605d732501e1"
Nov 02 12:34:04.773597 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.773533    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d395adc2e1c2293dd782593366d6e57e6dec93917a756c5734c605d732501e1"} err="failed to get container status \"1d395adc2e1c2293dd782593366d6e57e6dec93917a756c5734c605d732501e1\": rpc error: code = NotFound desc = could not find container \"1d395adc2e1c2293dd782593366d6e57e6dec93917a756c5734c605d732501e1\": container with ID starting with 1d395adc2e1c2293dd782593366d6e57e6dec93917a756c5734c605d732501e1 not found: ID does not exist"
Nov 02 12:34:04.773597 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.773555    4464 scope.go:122] "RemoveContainer" containerID="886d0c3fe1f711852cb41aa5416170ccc082175ccb5d914f176925e7349f7fcd"
Nov 02 12:34:04.773793 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:34:04.773772    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"886d0c3fe1f711852cb41aa5416170ccc082175ccb5d914f176925e7349f7fcd\": container with ID starting with 886d0c3fe1f711852cb41aa5416170ccc082175ccb5d914f176925e7349f7fcd not found: ID does not exist" containerID="886d0c3fe1f711852cb41aa5416170ccc082175ccb5d914f176925e7349f7fcd"
Nov 02 12:34:04.773836 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.773800    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886d0c3fe1f711852cb41aa5416170ccc082175ccb5d914f176925e7349f7fcd"} err="failed to get container status \"886d0c3fe1f711852cb41aa5416170ccc082175ccb5d914f176925e7349f7fcd\": rpc error: code = NotFound desc = could not find container \"886d0c3fe1f711852cb41aa5416170ccc082175ccb5d914f176925e7349f7fcd\": container with ID starting with 886d0c3fe1f711852cb41aa5416170ccc082175ccb5d914f176925e7349f7fcd not found: ID does not exist"
Nov 02 12:34:04.906518 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.906487    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"]
Nov 02 12:34:04.911046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:04.911019    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-q9k87"]
Nov 02 12:34:05.022146 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:05.022049    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"]
Nov 02 12:34:05.026299 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:05.026276    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-m89q5"]
Nov 02 12:34:05.090715 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:34:05.090687    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ab1ff_866c_4d73_bccb_7eded744c791.slice/crio-conmon-5587c946fad483f440f7f4ccc94254ed2a658671f453c373d06e24b90fbd7b7c.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:34:05.581436 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:05.581407    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3befcd84-95ad-4eed-88ad-eb6d332a2048" path="/var/lib/kubelet/pods/3befcd84-95ad-4eed-88ad-eb6d332a2048/volumes"
Nov 02 12:34:05.581863 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:05.581849    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" path="/var/lib/kubelet/pods/5d29abcb-5367-49e7-b4b0-4c27e87fe2a7/volumes"
Nov 02 12:34:07.575247 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:07.575218    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"]
Nov 02 12:34:07.575615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:07.575510    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j" podUID="a7a7742a-f60b-4270-8992-2855b79bed42" containerName="istio-proxy" containerID="cri-o://f6604e285d546b6f3d4319328b477b84071373fbb490ef3118e000161cb59194" gracePeriod=30
Nov 02 12:34:07.967281 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:34:07.967200    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ab1ff_866c_4d73_bccb_7eded744c791.slice/crio-conmon-5587c946fad483f440f7f4ccc94254ed2a658671f453c373d06e24b90fbd7b7c.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:34:08.442386 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:08.442352    4464 patch_prober.go:29] interesting pod/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:34:08.442652 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:08.442417    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j" podUID="a7a7742a-f60b-4270-8992-2855b79bed42" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:34:12.800361 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.800328    4464 generic.go:356] "Generic (PLEG): container finished" podID="a7a7742a-f60b-4270-8992-2855b79bed42" containerID="f6604e285d546b6f3d4319328b477b84071373fbb490ef3118e000161cb59194" exitCode=0
Nov 02 12:34:12.800827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.800458    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j" event={"ID":"a7a7742a-f60b-4270-8992-2855b79bed42","Type":"ContainerDied","Data":"f6604e285d546b6f3d4319328b477b84071373fbb490ef3118e000161cb59194"}
Nov 02 12:34:12.800827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.800543    4464 scope.go:122] "RemoveContainer" containerID="f81cb9696d14ff74d92bc0b637701470e44d76acca6b095b50336d6f61106064"
Nov 02 12:34:12.848140 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.848089    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:34:12.947799 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.947768    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-workload-certs\") pod \"a7a7742a-f60b-4270-8992-2855b79bed42\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") "
Nov 02 12:34:12.947912 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.947819    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-istio-envoy\") pod \"a7a7742a-f60b-4270-8992-2855b79bed42\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") "
Nov 02 12:34:12.947912 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.947866    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-istio-data\") pod \"a7a7742a-f60b-4270-8992-2855b79bed42\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") "
Nov 02 12:34:12.948035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.947908    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7a7742a-f60b-4270-8992-2855b79bed42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a7742a-f60b-4270-8992-2855b79bed42-istiod-ca-cert\") pod \"a7a7742a-f60b-4270-8992-2855b79bed42\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") "
Nov 02 12:34:12.948035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.947964    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a7742a-f60b-4270-8992-2855b79bed42-kube-api-access-zsrmz\" (UniqueName: \"kubernetes.io/projected/a7a7742a-f60b-4270-8992-2855b79bed42-kube-api-access-zsrmz\") pod \"a7a7742a-f60b-4270-8992-2855b79bed42\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") "
Nov 02 12:34:12.948143 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.948039    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a7742a-f60b-4270-8992-2855b79bed42-istio-token\" (UniqueName: \"kubernetes.io/projected/a7a7742a-f60b-4270-8992-2855b79bed42-istio-token\") pod \"a7a7742a-f60b-4270-8992-2855b79bed42\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") "
Nov 02 12:34:12.948143 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.948038    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-workload-certs" pod "a7a7742a-f60b-4270-8992-2855b79bed42" (UID: "a7a7742a-f60b-4270-8992-2855b79bed42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:12.948143 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.948068    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-credential-socket\") pod \"a7a7742a-f60b-4270-8992-2855b79bed42\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") "
Nov 02 12:34:12.948143 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.948100    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-workload-socket\") pod \"a7a7742a-f60b-4270-8992-2855b79bed42\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") "
Nov 02 12:34:12.948363 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.948235    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7a7742a-f60b-4270-8992-2855b79bed42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a7742a-f60b-4270-8992-2855b79bed42-istio-podinfo\") pod \"a7a7742a-f60b-4270-8992-2855b79bed42\" (UID: \"a7a7742a-f60b-4270-8992-2855b79bed42\") "
Nov 02 12:34:12.948363 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.948222    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-istio-data" pod "a7a7742a-f60b-4270-8992-2855b79bed42" (UID: "a7a7742a-f60b-4270-8992-2855b79bed42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:12.948481 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.948395    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-credential-socket" pod "a7a7742a-f60b-4270-8992-2855b79bed42" (UID: "a7a7742a-f60b-4270-8992-2855b79bed42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:12.948481 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.948420    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-workload-socket" pod "a7a7742a-f60b-4270-8992-2855b79bed42" (UID: "a7a7742a-f60b-4270-8992-2855b79bed42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:12.948481 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.948456    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a7742a-f60b-4270-8992-2855b79bed42-istiod-ca-cert" pod "a7a7742a-f60b-4270-8992-2855b79bed42" (UID: "a7a7742a-f60b-4270-8992-2855b79bed42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:34:12.948748 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.948730    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:12.948817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.948754    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:12.948817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.948769    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7a7742a-f60b-4270-8992-2855b79bed42-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:12.948817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.948783    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:12.948817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.948796    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:12.950023 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.949992    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-istio-envoy" pod "a7a7742a-f60b-4270-8992-2855b79bed42" (UID: "a7a7742a-f60b-4270-8992-2855b79bed42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:12.950118 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.950094    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7a7742a-f60b-4270-8992-2855b79bed42-istio-podinfo" pod "a7a7742a-f60b-4270-8992-2855b79bed42" (UID: "a7a7742a-f60b-4270-8992-2855b79bed42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:34:12.950482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.950468    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a7742a-f60b-4270-8992-2855b79bed42-kube-api-access-zsrmz" pod "a7a7742a-f60b-4270-8992-2855b79bed42" (UID: "a7a7742a-f60b-4270-8992-2855b79bed42"). InnerVolumeSpecName "kube-api-access-zsrmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:34:12.950523 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:12.950488    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a7742a-f60b-4270-8992-2855b79bed42-istio-token" pod "a7a7742a-f60b-4270-8992-2855b79bed42" (UID: "a7a7742a-f60b-4270-8992-2855b79bed42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:34:13.050304 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:13.050270    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7a7742a-f60b-4270-8992-2855b79bed42-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:13.050304 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:13.050298    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7a7742a-f60b-4270-8992-2855b79bed42-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:13.050304 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:13.050308    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7a7742a-f60b-4270-8992-2855b79bed42-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:13.050614 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:13.050317    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsrmz\" (UniqueName: \"kubernetes.io/projected/a7a7742a-f60b-4270-8992-2855b79bed42-kube-api-access-zsrmz\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:13.513690 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:13.513660    4464 scope.go:122] "RemoveContainer" containerID="b1efd3a785f50b8239538001c26a619c0268474b93f4404e765a249e37615c2b"
Nov 02 12:34:13.525967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:13.525944    4464 scope.go:122] "RemoveContainer" containerID="19259695bd0f46253a252230afd5bd1106c82859210318787c35376ad7c917f4"
Nov 02 12:34:13.534734 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:13.534710    4464 scope.go:122] "RemoveContainer" containerID="9341559b37be17df1785ca6e305c61bcdc179e2ec3c098d0d67f64a5b8e0d7f9"
Nov 02 12:34:13.543113 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:13.543069    4464 scope.go:122] "RemoveContainer" containerID="8fa026c52f3d7bc86ac278d184bbd2723445690e6cce34aabc45097222ddef16"
Nov 02 12:34:13.551072 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:13.551057    4464 scope.go:122] "RemoveContainer" containerID="92dca447239fc280ecbf97f56f00a390a9397abe43c7176708b82a6acbce7eda"
Nov 02 12:34:13.808375 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:13.808283    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j" event={"ID":"a7a7742a-f60b-4270-8992-2855b79bed42","Type":"ContainerDied","Data":"93a02fd2e749581edda2cf619bc20d8f335479f1b3af458338115bbd86a4fbb7"}
Nov 02 12:34:13.808375 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:13.808311    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"
Nov 02 12:34:13.808375 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:13.808342    4464 scope.go:122] "RemoveContainer" containerID="f6604e285d546b6f3d4319328b477b84071373fbb490ef3118e000161cb59194"
Nov 02 12:34:14.030226 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:14.030194    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"]
Nov 02 12:34:14.035939 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:14.035914    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-cf0095-openshift-default-66557c8649-rjm2j"]
Nov 02 12:34:15.050883 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:15.050848    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"]
Nov 02 12:34:15.101461 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:15.101427    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3d81c8b-sbx6n"]
Nov 02 12:34:15.503932 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:15.503896    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:34:15.581657 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:15.581612    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66d7c4c4-550a-4d5d-a16e-c0a554bce3f3" path="/var/lib/kubelet/pods/66d7c4c4-550a-4d5d-a16e-c0a554bce3f3/volumes"
Nov 02 12:34:15.582170 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:15.582149    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7a7742a-f60b-4270-8992-2855b79bed42" path="/var/lib/kubelet/pods/a7a7742a-f60b-4270-8992-2855b79bed42/volumes"
Nov 02 12:34:17.237075 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.237026    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"]
Nov 02 12:34:17.237661 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.237644    4464 cpu_manager.go:414] "containerMap: removing container" podUID="a7a7742a-f60b-4270-8992-2855b79bed42" containerName="istio-proxy"
Nov 02 12:34:17.237706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.237662    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7a7742a-f60b-4270-8992-2855b79bed42" containerName="istio-proxy"
Nov 02 12:34:17.237706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.237679    4464 cpu_manager.go:414] "containerMap: removing container" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerName="istio-proxy"
Nov 02 12:34:17.237706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.237685    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerName="istio-proxy"
Nov 02 12:34:17.237706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.237696    4464 cpu_manager.go:414] "containerMap: removing container" podUID="a7a7742a-f60b-4270-8992-2855b79bed42" containerName="istio-proxy"
Nov 02 12:34:17.237706 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.237701    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7a7742a-f60b-4270-8992-2855b79bed42" containerName="istio-proxy"
Nov 02 12:34:17.237864 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.237711    4464 cpu_manager.go:414] "containerMap: removing container" podUID="3befcd84-95ad-4eed-88ad-eb6d332a2048" containerName="istio-proxy"
Nov 02 12:34:17.237864 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.237716    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3befcd84-95ad-4eed-88ad-eb6d332a2048" containerName="istio-proxy"
Nov 02 12:34:17.237864 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.237728    4464 cpu_manager.go:414] "containerMap: removing container" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerName="istio-proxy"
Nov 02 12:34:17.237864 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.237733    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerName="istio-proxy"
Nov 02 12:34:17.237864 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.237813    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7a7742a-f60b-4270-8992-2855b79bed42" containerName="istio-proxy"
Nov 02 12:34:17.237864 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.237821    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerName="istio-proxy"
Nov 02 12:34:17.237864 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.237828    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7a7742a-f60b-4270-8992-2855b79bed42" containerName="istio-proxy"
Nov 02 12:34:17.237864 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.237835    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="3befcd84-95ad-4eed-88ad-eb6d332a2048" containerName="istio-proxy"
Nov 02 12:34:17.237864 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.237842    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d29abcb-5367-49e7-b4b0-4c27e87fe2a7" containerName="istio-proxy"
Nov 02 12:34:17.243240 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.243221    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"
Nov 02 12:34:17.299179 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.299152    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"]
Nov 02 12:34:17.351330 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.351295    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-6373b9-dockercfg-jn2d5\""
Nov 02 12:34:17.380998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.351637    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-6373b9\""
Nov 02 12:34:17.399472 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.399435    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f6b2381-3780-4856-8418-f55f4ac6e082-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk\" (UID: \"6f6b2381-3780-4856-8418-f55f4ac6e082\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"
Nov 02 12:34:17.399812 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.399791    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-997gg\" (UniqueName: \"kubernetes.io/projected/6f6b2381-3780-4856-8418-f55f4ac6e082-kube-api-access-997gg\") pod \"payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk\" (UID: \"6f6b2381-3780-4856-8418-f55f4ac6e082\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"
Nov 02 12:34:17.413222 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.413190    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"]
Nov 02 12:34:17.418503 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.418481    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"
Nov 02 12:34:17.442204 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.442139    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"]
Nov 02 12:34:17.501599 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.501508    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f6b2381-3780-4856-8418-f55f4ac6e082-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk\" (UID: \"6f6b2381-3780-4856-8418-f55f4ac6e082\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"
Nov 02 12:34:17.501778 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.501622    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-997gg\" (UniqueName: \"kubernetes.io/projected/6f6b2381-3780-4856-8418-f55f4ac6e082-kube-api-access-997gg\") pod \"payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk\" (UID: \"6f6b2381-3780-4856-8418-f55f4ac6e082\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"
Nov 02 12:34:17.501778 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.501703    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfcwq\" (UniqueName: \"kubernetes.io/projected/b2b1929e-1a37-4829-aa6b-2cc7aef86f0f-kube-api-access-rfcwq\") pod \"payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd\" (UID: \"b2b1929e-1a37-4829-aa6b-2cc7aef86f0f\") " pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"
Nov 02 12:34:17.501879 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.501842    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2b1929e-1a37-4829-aa6b-2cc7aef86f0f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd\" (UID: \"b2b1929e-1a37-4829-aa6b-2cc7aef86f0f\") " pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"
Nov 02 12:34:17.502264 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.502247    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f6b2381-3780-4856-8418-f55f4ac6e082-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk\" (UID: \"6f6b2381-3780-4856-8418-f55f4ac6e082\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"
Nov 02 12:34:17.563624 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.563592    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-997gg\" (UniqueName: \"kubernetes.io/projected/6f6b2381-3780-4856-8418-f55f4ac6e082-kube-api-access-997gg\") pod \"payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk\" (UID: \"6f6b2381-3780-4856-8418-f55f4ac6e082\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"
Nov 02 12:34:17.602394 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.602355    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2b1929e-1a37-4829-aa6b-2cc7aef86f0f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd\" (UID: \"b2b1929e-1a37-4829-aa6b-2cc7aef86f0f\") " pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"
Nov 02 12:34:17.602543 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.602488    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfcwq\" (UniqueName: \"kubernetes.io/projected/b2b1929e-1a37-4829-aa6b-2cc7aef86f0f-kube-api-access-rfcwq\") pod \"payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd\" (UID: \"b2b1929e-1a37-4829-aa6b-2cc7aef86f0f\") " pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"
Nov 02 12:34:17.603076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.603058    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2b1929e-1a37-4829-aa6b-2cc7aef86f0f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd\" (UID: \"b2b1929e-1a37-4829-aa6b-2cc7aef86f0f\") " pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"
Nov 02 12:34:17.664472 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.664444    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfcwq\" (UniqueName: \"kubernetes.io/projected/b2b1929e-1a37-4829-aa6b-2cc7aef86f0f-kube-api-access-rfcwq\") pod \"payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd\" (UID: \"b2b1929e-1a37-4829-aa6b-2cc7aef86f0f\") " pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"
Nov 02 12:34:17.743162 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.743133    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"
Nov 02 12:34:17.857289 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:17.857253    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"
Nov 02 12:34:18.120747 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:18.120718    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"]
Nov 02 12:34:18.121128 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:34:18.121101    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b1929e_1a37_4829_aa6b_2cc7aef86f0f.slice/crio-fde590a5a5158d3af0ab6c2eca6adea202c59f5198b4d00b43447c8a67d08197 WatchSource:0}: Error finding container fde590a5a5158d3af0ab6c2eca6adea202c59f5198b4d00b43447c8a67d08197: Status 404 returned error can't find the container with id fde590a5a5158d3af0ab6c2eca6adea202c59f5198b4d00b43447c8a67d08197
Nov 02 12:34:18.232755 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:18.232729    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"]
Nov 02 12:34:18.233512 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:34:18.233480    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6b2381_3780_4856_8418_f55f4ac6e082.slice/crio-e23c0d9cc47d2f5657febe50850b8717108897ae90f80bb40c9fbb01df9d57ec WatchSource:0}: Error finding container e23c0d9cc47d2f5657febe50850b8717108897ae90f80bb40c9fbb01df9d57ec: Status 404 returned error can't find the container with id e23c0d9cc47d2f5657febe50850b8717108897ae90f80bb40c9fbb01df9d57ec
Nov 02 12:34:18.843054 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:18.843011    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk" event={"ID":"6f6b2381-3780-4856-8418-f55f4ac6e082","Type":"ContainerStarted","Data":"fdfb5b099f0e854fb62f2d0d20572fabba89d30ffabf63010f67e9e6a06f2a5c"}
Nov 02 12:34:18.843488 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:18.843111    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"
Nov 02 12:34:18.843488 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:18.843132    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk" event={"ID":"6f6b2381-3780-4856-8418-f55f4ac6e082","Type":"ContainerStarted","Data":"e23c0d9cc47d2f5657febe50850b8717108897ae90f80bb40c9fbb01df9d57ec"}
Nov 02 12:34:18.844640 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:18.844614    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd" event={"ID":"b2b1929e-1a37-4829-aa6b-2cc7aef86f0f","Type":"ContainerStarted","Data":"39a310ee72f2a2d933225a36efd41df3501616e8747299288e395bac485da8a7"}
Nov 02 12:34:18.844764 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:18.844659    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd" event={"ID":"b2b1929e-1a37-4829-aa6b-2cc7aef86f0f","Type":"ContainerStarted","Data":"fde590a5a5158d3af0ab6c2eca6adea202c59f5198b4d00b43447c8a67d08197"}
Nov 02 12:34:18.844764 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:18.844746    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"
Nov 02 12:34:18.960126 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:18.960069    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk" podStartSLOduration=1.9600534440000001 podStartE2EDuration="1.960053444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:18.959527183 +0000 UTC m=+2287.523235593" watchObservedRunningTime="2026-11-02 12:34:18.960053444 +0000 UTC m=+2287.523761865"
Nov 02 12:34:23.088905 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:23.088848    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd" podStartSLOduration=6.088832748 podStartE2EDuration="6.088832748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:19.100013107 +0000 UTC m=+2287.663721518" watchObservedRunningTime="2026-11-02 12:34:23.088832748 +0000 UTC m=+2291.652541158"
Nov 02 12:34:23.090210 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:23.090182    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"]
Nov 02 12:34:28.091294 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:28.091206    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v" podUID="fcfaa90b-9297-4cfe-b835-29d2454322ac" containerName="payload-pre-processing" containerID="cri-o://c9342fe66fd039c2d55d437abc3acf94d8743b152a4a9c4a60371227ca95ef5e" gracePeriod=25
Nov 02 12:34:28.299589 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:34:28.299550    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:34:28.349839 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:28.349775    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"
Nov 02 12:34:28.387288 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:28.387262    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcfaa90b-9297-4cfe-b835-29d2454322ac-kube-api-access-rc6wk\" (UniqueName: \"kubernetes.io/projected/fcfaa90b-9297-4cfe-b835-29d2454322ac-kube-api-access-rc6wk\") pod \"fcfaa90b-9297-4cfe-b835-29d2454322ac\" (UID: \"fcfaa90b-9297-4cfe-b835-29d2454322ac\") "
Nov 02 12:34:28.387427 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:28.387333    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcfaa90b-9297-4cfe-b835-29d2454322ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfaa90b-9297-4cfe-b835-29d2454322ac-plugins-config-volume\") pod \"fcfaa90b-9297-4cfe-b835-29d2454322ac\" (UID: \"fcfaa90b-9297-4cfe-b835-29d2454322ac\") "
Nov 02 12:34:28.387799 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:28.387768    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfaa90b-9297-4cfe-b835-29d2454322ac-plugins-config-volume" pod "fcfaa90b-9297-4cfe-b835-29d2454322ac" (UID: "fcfaa90b-9297-4cfe-b835-29d2454322ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:34:28.389283 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:28.389264    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfaa90b-9297-4cfe-b835-29d2454322ac-kube-api-access-rc6wk" pod "fcfaa90b-9297-4cfe-b835-29d2454322ac" (UID: "fcfaa90b-9297-4cfe-b835-29d2454322ac"). InnerVolumeSpecName "kube-api-access-rc6wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:34:28.488927 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:28.488895    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc6wk\" (UniqueName: \"kubernetes.io/projected/fcfaa90b-9297-4cfe-b835-29d2454322ac-kube-api-access-rc6wk\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:28.488927 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:28.488922    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfaa90b-9297-4cfe-b835-29d2454322ac-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:28.905711 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:28.905674    4464 generic.go:356] "Generic (PLEG): container finished" podID="fcfaa90b-9297-4cfe-b835-29d2454322ac" containerID="c9342fe66fd039c2d55d437abc3acf94d8743b152a4a9c4a60371227ca95ef5e" exitCode=0
Nov 02 12:34:28.905861 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:28.905721    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"
Nov 02 12:34:28.905861 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:28.905724    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v" event={"ID":"fcfaa90b-9297-4cfe-b835-29d2454322ac","Type":"ContainerDied","Data":"c9342fe66fd039c2d55d437abc3acf94d8743b152a4a9c4a60371227ca95ef5e"}
Nov 02 12:34:28.905861 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:28.905774    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v" event={"ID":"fcfaa90b-9297-4cfe-b835-29d2454322ac","Type":"ContainerDied","Data":"56be09a043534002bff16fbaaa2f2445b48a432c88e40e1edd3f2a4842c18846"}
Nov 02 12:34:28.905861 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:28.905796    4464 scope.go:122] "RemoveContainer" containerID="c9342fe66fd039c2d55d437abc3acf94d8743b152a4a9c4a60371227ca95ef5e"
Nov 02 12:34:28.919258 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:28.919240    4464 scope.go:122] "RemoveContainer" containerID="c9342fe66fd039c2d55d437abc3acf94d8743b152a4a9c4a60371227ca95ef5e"
Nov 02 12:34:28.919527 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:34:28.919509    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9342fe66fd039c2d55d437abc3acf94d8743b152a4a9c4a60371227ca95ef5e\": container with ID starting with c9342fe66fd039c2d55d437abc3acf94d8743b152a4a9c4a60371227ca95ef5e not found: ID does not exist" containerID="c9342fe66fd039c2d55d437abc3acf94d8743b152a4a9c4a60371227ca95ef5e"
Nov 02 12:34:28.919589 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:28.919534    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9342fe66fd039c2d55d437abc3acf94d8743b152a4a9c4a60371227ca95ef5e"} err="failed to get container status \"c9342fe66fd039c2d55d437abc3acf94d8743b152a4a9c4a60371227ca95ef5e\": rpc error: code = NotFound desc = could not find container \"c9342fe66fd039c2d55d437abc3acf94d8743b152a4a9c4a60371227ca95ef5e\": container with ID starting with c9342fe66fd039c2d55d437abc3acf94d8743b152a4a9c4a60371227ca95ef5e not found: ID does not exist"
Nov 02 12:34:29.074437 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:29.074398    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"]
Nov 02 12:34:29.129053 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:29.129014    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2e101a-68f7d76dc9-cl42v"]
Nov 02 12:34:29.581594 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:29.581558    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcfaa90b-9297-4cfe-b835-29d2454322ac" path="/var/lib/kubelet/pods/fcfaa90b-9297-4cfe-b835-29d2454322ac/volumes"
Nov 02 12:34:29.861134 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:29.860585    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"
Nov 02 12:34:29.879943 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:29.879911    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"
Nov 02 12:34:35.124624 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:35.124583    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:34:36.772193 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:36.772156    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"]
Nov 02 12:34:36.772701 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:36.772420    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q" podUID="51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" containerName="istio-proxy" containerID="cri-o://c3d50eacb69a9f21ec1d3260fd170832d0f2af2e1d1600855d97901276a38072" gracePeriod=30
Nov 02 12:34:38.912569 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:38.912532    4464 patch_prober.go:29] interesting pod/e2e-api-a-2e101a-openshift-default-88548686f-drm8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:34:38.912969 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:38.912600    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q" podUID="51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:34:42.050855 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.050782    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"]
Nov 02 12:34:42.060113 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.060082    4464 generic.go:356] "Generic (PLEG): container finished" podID="51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" containerID="c3d50eacb69a9f21ec1d3260fd170832d0f2af2e1d1600855d97901276a38072" exitCode=0
Nov 02 12:34:42.060220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.060157    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q" event={"ID":"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e","Type":"ContainerDied","Data":"c3d50eacb69a9f21ec1d3260fd170832d0f2af2e1d1600855d97901276a38072"}
Nov 02 12:34:42.060267 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.060230    4464 scope.go:122] "RemoveContainer" containerID="cdf627f5ce63cf92589fd95c214f9c3a27a04658d89262ebe91bb9d763249a51"
Nov 02 12:34:42.085211 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.085191    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:34:42.101156 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.101133    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30bd0275-47hsq"]
Nov 02 12:34:42.190338 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.190307    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-envoy\") pod \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") "
Nov 02 12:34:42.190477 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.190367    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istiod-ca-cert\") pod \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") "
Nov 02 12:34:42.190477 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.190396    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-podinfo\") pod \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") "
Nov 02 12:34:42.190477 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.190443    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-token\" (UniqueName: \"kubernetes.io/projected/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-token\") pod \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") "
Nov 02 12:34:42.190635 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.190474    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-kube-api-access-sgbh6\" (UniqueName: \"kubernetes.io/projected/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-kube-api-access-sgbh6\") pod \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") "
Nov 02 12:34:42.190697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.190643    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-credential-socket\") pod \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") "
Nov 02 12:34:42.190697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.190683    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-workload-socket\") pod \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") "
Nov 02 12:34:42.190811 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.190789    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-data\") pod \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") "
Nov 02 12:34:42.190872 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.190807    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istiod-ca-cert" pod "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" (UID: "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:34:42.190872 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.190822    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-workload-certs\") pod \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\" (UID: \"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e\") "
Nov 02 12:34:42.191067 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.190936    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-credential-socket" pod "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" (UID: "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:42.191152 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.191101    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-workload-socket" pod "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" (UID: "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:42.191215 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.191141    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-data" pod "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" (UID: "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:42.191215 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.191153    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-workload-certs" pod "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" (UID: "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:42.191368 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.191350    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:42.191462 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.191375    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:42.191462 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.191390    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:42.191462 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.191404    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:42.191462 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.191418    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:42.192794 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.192763    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-token" pod "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" (UID: "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:34:42.192907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.192882    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-podinfo" pod "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" (UID: "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:34:42.193037 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.193020    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-kube-api-access-sgbh6" pod "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" (UID: "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e"). InnerVolumeSpecName "kube-api-access-sgbh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:34:42.193106 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.193045    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-envoy" pod "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" (UID: "51d041eb-5ae8-46ea-b1cc-cebfa1973a1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:34:42.292166 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.292138    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:42.292166 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.292162    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:42.292166 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.292173    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:42.292400 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:42.292183    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgbh6\" (UniqueName: \"kubernetes.io/projected/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e-kube-api-access-sgbh6\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:43.067751 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:43.067713    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q" event={"ID":"51d041eb-5ae8-46ea-b1cc-cebfa1973a1e","Type":"ContainerDied","Data":"b538bf495c86c263a1cda1ecf658c324c4cec18ae0f64fdabce817377c16690e"}
Nov 02 12:34:43.067751 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:43.067750    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"
Nov 02 12:34:43.068299 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:43.067775    4464 scope.go:122] "RemoveContainer" containerID="c3d50eacb69a9f21ec1d3260fd170832d0f2af2e1d1600855d97901276a38072"
Nov 02 12:34:43.357172 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:43.357101    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"]
Nov 02 12:34:43.361263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:43.361239    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2e101a-openshift-default-88548686f-drm8q"]
Nov 02 12:34:43.582106 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:43.582070    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" path="/var/lib/kubelet/pods/51d041eb-5ae8-46ea-b1cc-cebfa1973a1e/volumes"
Nov 02 12:34:43.582684 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:43.582664    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea" path="/var/lib/kubelet/pods/b9ae1bdd-cf69-4dcc-a2a6-6495667e0bea/volumes"
Nov 02 12:34:46.994039 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:46.994006    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:34:49.152353 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:49.152304    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"]
Nov 02 12:34:54.047527 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:54.047487    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"]
Nov 02 12:34:54.068583 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:54.068547    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1cd40b6-flc24"]
Nov 02 12:34:54.152671 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:54.152638    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn" podUID="c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5" containerName="payload-processing" containerID="cri-o://05f0a367b9b1faab6c898c6f8508516b096fc77e87650355ded89fc18364d2aa" gracePeriod=25
Nov 02 12:34:54.404268 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:54.404246    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"
Nov 02 12:34:54.555820 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:54.555791    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5-kube-api-access-hjdtn\" (UniqueName: \"kubernetes.io/projected/c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5-kube-api-access-hjdtn\") pod \"c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5\" (UID: \"c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5\") "
Nov 02 12:34:54.556032 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:54.555870    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5-plugins-config-volume\") pod \"c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5\" (UID: \"c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5\") "
Nov 02 12:34:54.556353 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:54.556326    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5-plugins-config-volume" pod "c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5" (UID: "c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:34:54.558046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:54.558017    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5-kube-api-access-hjdtn" pod "c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5" (UID: "c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5"). InnerVolumeSpecName "kube-api-access-hjdtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:34:54.657371 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:54.657295    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjdtn\" (UniqueName: \"kubernetes.io/projected/c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5-kube-api-access-hjdtn\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:54.657371 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:54.657334    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:34:55.145469 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:55.145438    4464 generic.go:356] "Generic (PLEG): container finished" podID="c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5" containerID="05f0a367b9b1faab6c898c6f8508516b096fc77e87650355ded89fc18364d2aa" exitCode=0
Nov 02 12:34:55.145469 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:55.145473    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn" event={"ID":"c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5","Type":"ContainerDied","Data":"05f0a367b9b1faab6c898c6f8508516b096fc77e87650355ded89fc18364d2aa"}
Nov 02 12:34:55.146024 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:55.145482    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"
Nov 02 12:34:55.146024 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:55.145517    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn" event={"ID":"c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5","Type":"ContainerDied","Data":"41dd71b6466aa9299fdf74bbee891a3df7dc06c90c3963d2ceda3cbc0651a108"}
Nov 02 12:34:55.146024 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:55.145538    4464 scope.go:122] "RemoveContainer" containerID="05f0a367b9b1faab6c898c6f8508516b096fc77e87650355ded89fc18364d2aa"
Nov 02 12:34:55.163822 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:55.163664    4464 scope.go:122] "RemoveContainer" containerID="05f0a367b9b1faab6c898c6f8508516b096fc77e87650355ded89fc18364d2aa"
Nov 02 12:34:55.163968 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:34:55.163948    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f0a367b9b1faab6c898c6f8508516b096fc77e87650355ded89fc18364d2aa\": container with ID starting with 05f0a367b9b1faab6c898c6f8508516b096fc77e87650355ded89fc18364d2aa not found: ID does not exist" containerID="05f0a367b9b1faab6c898c6f8508516b096fc77e87650355ded89fc18364d2aa"
Nov 02 12:34:55.164080 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:55.163991    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f0a367b9b1faab6c898c6f8508516b096fc77e87650355ded89fc18364d2aa"} err="failed to get container status \"05f0a367b9b1faab6c898c6f8508516b096fc77e87650355ded89fc18364d2aa\": rpc error: code = NotFound desc = could not find container \"05f0a367b9b1faab6c898c6f8508516b096fc77e87650355ded89fc18364d2aa\": container with ID starting with 05f0a367b9b1faab6c898c6f8508516b096fc77e87650355ded89fc18364d2aa not found: ID does not exist"
Nov 02 12:34:55.312069 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:55.312041    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"]
Nov 02 12:34:55.362438 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:55.362409    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-257e6d-86977c947f-t9chn"]
Nov 02 12:34:55.581166 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:55.581138    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5" path="/var/lib/kubelet/pods/c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5/volumes"
Nov 02 12:34:55.581551 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:34:55.581539    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f09f75fe-08ae-4c85-9bbc-62aae816d7c3" path="/var/lib/kubelet/pods/f09f75fe-08ae-4c85-9bbc-62aae816d7c3/volumes"
Nov 02 12:34:58.291741 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:34:58.291704    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:35:00.047954 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:00.047923    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"]
Nov 02 12:35:00.098351 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:00.098311    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca95735-ctrmf"]
Nov 02 12:35:01.581369 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:01.581335    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e017e5c3-9876-47ff-8c33-2d53501e03dd" path="/var/lib/kubelet/pods/e017e5c3-9876-47ff-8c33-2d53501e03dd/volumes"
Nov 02 12:35:05.998012 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:05.997961    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:35:13.616125 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:13.616094    4464 scope.go:122] "RemoveContainer" containerID="37a9018dcb72abdff8696bee036d1fe206f74d772a6e4d722e2ff5061eea8fa5"
Nov 02 12:35:13.625263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:13.625243    4464 scope.go:122] "RemoveContainer" containerID="e7e74b79b565a83946ded02528077c5070dda02980499d758599e6237232a97c"
Nov 02 12:35:25.064898 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:25.064865    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:35:28.291557 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:35:28.291517    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:35:30.048331 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:30.048299    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"]
Nov 02 12:35:30.098654 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:30.098620    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc460cdc-7rbq8"]
Nov 02 12:35:31.588524 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:31.588491    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a7bd74-36f4-4974-975a-506f257b6a60" path="/var/lib/kubelet/pods/54a7bd74-36f4-4974-975a-506f257b6a60/volumes"
Nov 02 12:35:41.158558 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:41.158525    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"]
Nov 02 12:35:41.159008 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:41.158741    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp" podUID="09dda7c2-acb9-43ef-a3c2-e7147de8d3be" containerName="istio-proxy" containerID="cri-o://ac0bf84e548d8b4dac8619246abb0c9b741a77d84c9c4b394d9eae450217a967" gracePeriod=30
Nov 02 12:35:41.230923 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:41.230894    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:35:44.237156 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:44.237115    4464 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:35:44.237561 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:44.237185    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp" podUID="09dda7c2-acb9-43ef-a3c2-e7147de8d3be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:35:46.049876 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.049844    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"]
Nov 02 12:35:46.100222 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.100190    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a6e55f8-87lt7"]
Nov 02 12:35:46.415231 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.415204    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:35:46.443832 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.443804    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istiod-ca-cert\") pod \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") "
Nov 02 12:35:46.443962 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.443839    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-podinfo\") pod \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") "
Nov 02 12:35:46.443962 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.443882    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-envoy\") pod \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") "
Nov 02 12:35:46.443962 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.443928    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-workload-certs\") pod \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") "
Nov 02 12:35:46.443962 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.443957    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-kube-api-access-crscp\" (UniqueName: \"kubernetes.io/projected/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-kube-api-access-crscp\") pod \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") "
Nov 02 12:35:46.444195 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.444015    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-credential-socket\") pod \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") "
Nov 02 12:35:46.444195 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.444113    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-workload-socket\") pod \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") "
Nov 02 12:35:46.444195 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.444134    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-data\") pod \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") "
Nov 02 12:35:46.444378 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.444208    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-token\" (UniqueName: \"kubernetes.io/projected/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-token\") pod \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\" (UID: \"09dda7c2-acb9-43ef-a3c2-e7147de8d3be\") "
Nov 02 12:35:46.444561 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.444203    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istiod-ca-cert" pod "09dda7c2-acb9-43ef-a3c2-e7147de8d3be" (UID: "09dda7c2-acb9-43ef-a3c2-e7147de8d3be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:35:46.444668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.444229    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-workload-certs" pod "09dda7c2-acb9-43ef-a3c2-e7147de8d3be" (UID: "09dda7c2-acb9-43ef-a3c2-e7147de8d3be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:35:46.444668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.444464    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-workload-socket" pod "09dda7c2-acb9-43ef-a3c2-e7147de8d3be" (UID: "09dda7c2-acb9-43ef-a3c2-e7147de8d3be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:35:46.444668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.444529    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-credential-socket" pod "09dda7c2-acb9-43ef-a3c2-e7147de8d3be" (UID: "09dda7c2-acb9-43ef-a3c2-e7147de8d3be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:35:46.444840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.444732    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:35:46.444840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.444747    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:35:46.444840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.444756    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:35:46.444840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.444765    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:35:46.444840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.444778    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-data" pod "09dda7c2-acb9-43ef-a3c2-e7147de8d3be" (UID: "09dda7c2-acb9-43ef-a3c2-e7147de8d3be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:35:46.446087 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.446065    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-envoy" pod "09dda7c2-acb9-43ef-a3c2-e7147de8d3be" (UID: "09dda7c2-acb9-43ef-a3c2-e7147de8d3be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:35:46.446346 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.446325    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-kube-api-access-crscp" pod "09dda7c2-acb9-43ef-a3c2-e7147de8d3be" (UID: "09dda7c2-acb9-43ef-a3c2-e7147de8d3be"). InnerVolumeSpecName "kube-api-access-crscp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:35:46.446470 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.446445    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-podinfo" pod "09dda7c2-acb9-43ef-a3c2-e7147de8d3be" (UID: "09dda7c2-acb9-43ef-a3c2-e7147de8d3be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:35:46.446569 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.446469    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-token" pod "09dda7c2-acb9-43ef-a3c2-e7147de8d3be" (UID: "09dda7c2-acb9-43ef-a3c2-e7147de8d3be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:35:46.467664 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.467641    4464 generic.go:356] "Generic (PLEG): container finished" podID="09dda7c2-acb9-43ef-a3c2-e7147de8d3be" containerID="ac0bf84e548d8b4dac8619246abb0c9b741a77d84c9c4b394d9eae450217a967" exitCode=0
Nov 02 12:35:46.467764 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.467682    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"
Nov 02 12:35:46.467764 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.467721    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp" event={"ID":"09dda7c2-acb9-43ef-a3c2-e7147de8d3be","Type":"ContainerDied","Data":"ac0bf84e548d8b4dac8619246abb0c9b741a77d84c9c4b394d9eae450217a967"}
Nov 02 12:35:46.467838 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.467774    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp" event={"ID":"09dda7c2-acb9-43ef-a3c2-e7147de8d3be","Type":"ContainerDied","Data":"9e65077e9bac53fe81846cb6df8012eafed7df7a60d55b5070648edd58a33a7d"}
Nov 02 12:35:46.467838 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.467799    4464 scope.go:122] "RemoveContainer" containerID="ac0bf84e548d8b4dac8619246abb0c9b741a77d84c9c4b394d9eae450217a967"
Nov 02 12:35:46.477484 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.477469    4464 scope.go:122] "RemoveContainer" containerID="ac0bf84e548d8b4dac8619246abb0c9b741a77d84c9c4b394d9eae450217a967"
Nov 02 12:35:46.477746 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:35:46.477726    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0bf84e548d8b4dac8619246abb0c9b741a77d84c9c4b394d9eae450217a967\": container with ID starting with ac0bf84e548d8b4dac8619246abb0c9b741a77d84c9c4b394d9eae450217a967 not found: ID does not exist" containerID="ac0bf84e548d8b4dac8619246abb0c9b741a77d84c9c4b394d9eae450217a967"
Nov 02 12:35:46.477811 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.477755    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0bf84e548d8b4dac8619246abb0c9b741a77d84c9c4b394d9eae450217a967"} err="failed to get container status \"ac0bf84e548d8b4dac8619246abb0c9b741a77d84c9c4b394d9eae450217a967\": rpc error: code = NotFound desc = could not find container \"ac0bf84e548d8b4dac8619246abb0c9b741a77d84c9c4b394d9eae450217a967\": container with ID starting with ac0bf84e548d8b4dac8619246abb0c9b741a77d84c9c4b394d9eae450217a967 not found: ID does not exist"
Nov 02 12:35:46.546202 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.546173    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:35:46.546202 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.546201    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crscp\" (UniqueName: \"kubernetes.io/projected/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-kube-api-access-crscp\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:35:46.546387 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.546215    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:35:46.546387 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.546226    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:35:46.546387 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.546237    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09dda7c2-acb9-43ef-a3c2-e7147de8d3be-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:35:46.659803 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.659720    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"]
Nov 02 12:35:46.665942 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:46.665915    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cr2cp"]
Nov 02 12:35:47.582203 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:47.582168    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09dda7c2-acb9-43ef-a3c2-e7147de8d3be" path="/var/lib/kubelet/pods/09dda7c2-acb9-43ef-a3c2-e7147de8d3be/volumes"
Nov 02 12:35:47.582598 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:47.582581    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75611cca-49e2-4172-8d37-089b6c0295dd" path="/var/lib/kubelet/pods/75611cca-49e2-4172-8d37-089b6c0295dd/volumes"
Nov 02 12:35:54.729252 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:35:54.729167    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:35:58.291421 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:35:58.291384    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:36:07.882372 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:07.882339    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"]
Nov 02 12:36:07.882814 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:07.882529    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh" podUID="666a9bf6-5863-44ce-9574-26ca5ee9b856" containerName="istio-proxy" containerID="cri-o://8b7b6d5ee62e12ba0477f445fff4b9510c399976a5a8863d39bd85444af4d551" gracePeriod=30
Nov 02 12:36:11.299796 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:11.299760    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"]
Nov 02 12:36:11.300441 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:11.300023    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm" podUID="d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" containerName="istio-proxy" containerID="cri-o://ef10021c51e6b358bc480b49b9aa15e60f62df94bad10212eadbdeb94ea3686f" gracePeriod=30
Nov 02 12:36:11.350932 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:11.350896    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"]
Nov 02 12:36:11.351176 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:11.351152    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4" podUID="eb51d042-8930-45f9-89c9-fe9ad351df24" containerName="istio-proxy" containerID="cri-o://9cf4dc362091b40edd23666bb3978ed460ed494b6e14a13c5b60cdb2eb37a3e3" gracePeriod=30
Nov 02 12:36:11.353205 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:11.353179    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"]
Nov 02 12:36:11.353374 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:11.353355    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv" podUID="890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" containerName="istio-proxy" containerID="cri-o://f82b80bb5789034c6329a820dc78fb89a27d1b57da62eb9d50f37feb5629b1e5" gracePeriod=30
Nov 02 12:36:11.401654 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:11.401620    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"]
Nov 02 12:36:11.401823 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:11.401806    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c" podUID="896e3cac-30cf-492b-a20a-cec437dc6a05" containerName="istio-proxy" containerID="cri-o://419f3f75d7874891e9dd2af2149c76a2ce261aee46c0ece5e5f51d184547be5a" gracePeriod=30
Nov 02 12:36:11.916867 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:11.916842    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:36:11.934281 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:11.934257    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:36:11.952660 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:11.952639    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:36:11.970231 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:11.970214    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:36:12.897294 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:12.897259    4464 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 02 12:36:12.897602 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:12.897336    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh" podUID="666a9bf6-5863-44ce-9574-26ca5ee9b856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 02 12:36:13.132909 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.132887    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:36:13.226492 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.226468    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666a9bf6-5863-44ce-9574-26ca5ee9b856-kube-api-access-g8xzh\" (UniqueName: \"kubernetes.io/projected/666a9bf6-5863-44ce-9574-26ca5ee9b856-kube-api-access-g8xzh\") pod \"666a9bf6-5863-44ce-9574-26ca5ee9b856\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") "
Nov 02 12:36:13.226649 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.226509    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-podinfo\") pod \"666a9bf6-5863-44ce-9574-26ca5ee9b856\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") "
Nov 02 12:36:13.226649 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.226574    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-credential-socket\") pod \"666a9bf6-5863-44ce-9574-26ca5ee9b856\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") "
Nov 02 12:36:13.226649 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.226642    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-envoy\") pod \"666a9bf6-5863-44ce-9574-26ca5ee9b856\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") "
Nov 02 12:36:13.226767 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.226730    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/666a9bf6-5863-44ce-9574-26ca5ee9b856-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666a9bf6-5863-44ce-9574-26ca5ee9b856-istiod-ca-cert\") pod \"666a9bf6-5863-44ce-9574-26ca5ee9b856\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") "
Nov 02 12:36:13.226812 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.226790    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-data\") pod \"666a9bf6-5863-44ce-9574-26ca5ee9b856\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") "
Nov 02 12:36:13.226868 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.226822    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-workload-socket\") pod \"666a9bf6-5863-44ce-9574-26ca5ee9b856\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") "
Nov 02 12:36:13.226923 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.226880    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-workload-certs\") pod \"666a9bf6-5863-44ce-9574-26ca5ee9b856\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") "
Nov 02 12:36:13.226923 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.226908    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-token\" (UniqueName: \"kubernetes.io/projected/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-token\") pod \"666a9bf6-5863-44ce-9574-26ca5ee9b856\" (UID: \"666a9bf6-5863-44ce-9574-26ca5ee9b856\") "
Nov 02 12:36:13.227083 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.226927    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-credential-socket" pod "666a9bf6-5863-44ce-9574-26ca5ee9b856" (UID: "666a9bf6-5863-44ce-9574-26ca5ee9b856"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:13.227196 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.227174    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-workload-socket" pod "666a9bf6-5863-44ce-9574-26ca5ee9b856" (UID: "666a9bf6-5863-44ce-9574-26ca5ee9b856"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:13.227259 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.227204    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-workload-certs" pod "666a9bf6-5863-44ce-9574-26ca5ee9b856" (UID: "666a9bf6-5863-44ce-9574-26ca5ee9b856"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:13.227259 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.227218    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/666a9bf6-5863-44ce-9574-26ca5ee9b856-istiod-ca-cert" pod "666a9bf6-5863-44ce-9574-26ca5ee9b856" (UID: "666a9bf6-5863-44ce-9574-26ca5ee9b856"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:13.227354 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.227278    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-data" pod "666a9bf6-5863-44ce-9574-26ca5ee9b856" (UID: "666a9bf6-5863-44ce-9574-26ca5ee9b856"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:13.227480 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.227462    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666a9bf6-5863-44ce-9574-26ca5ee9b856-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:13.227556 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.227481    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:13.227556 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.227492    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:13.227556 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.227500    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:13.227556 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.227512    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:13.228712 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.228683    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666a9bf6-5863-44ce-9574-26ca5ee9b856-kube-api-access-g8xzh" pod "666a9bf6-5863-44ce-9574-26ca5ee9b856" (UID: "666a9bf6-5863-44ce-9574-26ca5ee9b856"). InnerVolumeSpecName "kube-api-access-g8xzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:13.229108 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.229082    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-token" pod "666a9bf6-5863-44ce-9574-26ca5ee9b856" (UID: "666a9bf6-5863-44ce-9574-26ca5ee9b856"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:13.229194 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.229171    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-envoy" pod "666a9bf6-5863-44ce-9574-26ca5ee9b856" (UID: "666a9bf6-5863-44ce-9574-26ca5ee9b856"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:13.229238 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.229173    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-podinfo" pod "666a9bf6-5863-44ce-9574-26ca5ee9b856" (UID: "666a9bf6-5863-44ce-9574-26ca5ee9b856"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:36:13.328535 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.328503    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8xzh\" (UniqueName: \"kubernetes.io/projected/666a9bf6-5863-44ce-9574-26ca5ee9b856-kube-api-access-g8xzh\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:13.328535 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.328533    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:13.328737 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.328549    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:13.328737 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.328561    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666a9bf6-5863-44ce-9574-26ca5ee9b856-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:13.624797 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.624764    4464 generic.go:356] "Generic (PLEG): container finished" podID="666a9bf6-5863-44ce-9574-26ca5ee9b856" containerID="8b7b6d5ee62e12ba0477f445fff4b9510c399976a5a8863d39bd85444af4d551" exitCode=0
Nov 02 12:36:13.624954 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.624808    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"
Nov 02 12:36:13.624954 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.624839    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh" event={"ID":"666a9bf6-5863-44ce-9574-26ca5ee9b856","Type":"ContainerDied","Data":"8b7b6d5ee62e12ba0477f445fff4b9510c399976a5a8863d39bd85444af4d551"}
Nov 02 12:36:13.624954 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.624891    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh" event={"ID":"666a9bf6-5863-44ce-9574-26ca5ee9b856","Type":"ContainerDied","Data":"1542f90e6705ad5954ea1f9a35f9295d064cfa097f9529b3aea8b2c61c428c4b"}
Nov 02 12:36:13.624954 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.624915    4464 scope.go:122] "RemoveContainer" containerID="8b7b6d5ee62e12ba0477f445fff4b9510c399976a5a8863d39bd85444af4d551"
Nov 02 12:36:13.637335 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.637285    4464 scope.go:122] "RemoveContainer" containerID="8b7b6d5ee62e12ba0477f445fff4b9510c399976a5a8863d39bd85444af4d551"
Nov 02 12:36:13.637746 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:36:13.637728    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b7b6d5ee62e12ba0477f445fff4b9510c399976a5a8863d39bd85444af4d551\": container with ID starting with 8b7b6d5ee62e12ba0477f445fff4b9510c399976a5a8863d39bd85444af4d551 not found: ID does not exist" containerID="8b7b6d5ee62e12ba0477f445fff4b9510c399976a5a8863d39bd85444af4d551"
Nov 02 12:36:13.637824 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.637758    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b7b6d5ee62e12ba0477f445fff4b9510c399976a5a8863d39bd85444af4d551"} err="failed to get container status \"8b7b6d5ee62e12ba0477f445fff4b9510c399976a5a8863d39bd85444af4d551\": rpc error: code = NotFound desc = could not find container \"8b7b6d5ee62e12ba0477f445fff4b9510c399976a5a8863d39bd85444af4d551\": container with ID starting with 8b7b6d5ee62e12ba0477f445fff4b9510c399976a5a8863d39bd85444af4d551 not found: ID does not exist"
Nov 02 12:36:13.668492 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.668458    4464 scope.go:122] "RemoveContainer" containerID="82c77c83ea8adc7c600be9f2048d13d8cb3f66052747417c25669ff4a705561c"
Nov 02 12:36:13.677445 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.677426    4464 scope.go:122] "RemoveContainer" containerID="0cfa22d252a6c0e3d0cb02c46e5235e27ba60440d44a2f66707fa13faec016b7"
Nov 02 12:36:13.685517 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.685498    4464 scope.go:122] "RemoveContainer" containerID="4f026e2914df704f2fbcfeaeb319eb14b291fda80e7117e737f49ed9382c08f0"
Nov 02 12:36:13.793899 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.793871    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"]
Nov 02 12:36:13.844201 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:13.844129    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-77kfh"]
Nov 02 12:36:15.582298 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:15.582266    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="666a9bf6-5863-44ce-9574-26ca5ee9b856" path="/var/lib/kubelet/pods/666a9bf6-5863-44ce-9574-26ca5ee9b856/volumes"
Nov 02 12:36:16.149159 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.149123    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lwhz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:36:16.149373 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.149176    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4" podUID="eb51d042-8930-45f9-89c9-fe9ad351df24" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:36:16.622284 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.622256    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:36:16.644321 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.644292    4464 generic.go:356] "Generic (PLEG): container finished" podID="890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" containerID="f82b80bb5789034c6329a820dc78fb89a27d1b57da62eb9d50f37feb5629b1e5" exitCode=0
Nov 02 12:36:16.644444 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.644367    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv" event={"ID":"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f","Type":"ContainerDied","Data":"f82b80bb5789034c6329a820dc78fb89a27d1b57da62eb9d50f37feb5629b1e5"}
Nov 02 12:36:16.646447 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.646413    4464 generic.go:356] "Generic (PLEG): container finished" podID="896e3cac-30cf-492b-a20a-cec437dc6a05" containerID="419f3f75d7874891e9dd2af2149c76a2ce261aee46c0ece5e5f51d184547be5a" exitCode=0
Nov 02 12:36:16.646571 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.646494    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c" event={"ID":"896e3cac-30cf-492b-a20a-cec437dc6a05","Type":"ContainerDied","Data":"419f3f75d7874891e9dd2af2149c76a2ce261aee46c0ece5e5f51d184547be5a"}
Nov 02 12:36:16.648232 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.648212    4464 generic.go:356] "Generic (PLEG): container finished" podID="eb51d042-8930-45f9-89c9-fe9ad351df24" containerID="9cf4dc362091b40edd23666bb3978ed460ed494b6e14a13c5b60cdb2eb37a3e3" exitCode=0
Nov 02 12:36:16.648341 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.648283    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4" event={"ID":"eb51d042-8930-45f9-89c9-fe9ad351df24","Type":"ContainerDied","Data":"9cf4dc362091b40edd23666bb3978ed460ed494b6e14a13c5b60cdb2eb37a3e3"}
Nov 02 12:36:16.650132 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.649671    4464 generic.go:356] "Generic (PLEG): container finished" podID="d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" containerID="ef10021c51e6b358bc480b49b9aa15e60f62df94bad10212eadbdeb94ea3686f" exitCode=0
Nov 02 12:36:16.650132 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.649733    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"
Nov 02 12:36:16.650132 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.649770    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm" event={"ID":"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25","Type":"ContainerDied","Data":"ef10021c51e6b358bc480b49b9aa15e60f62df94bad10212eadbdeb94ea3686f"}
Nov 02 12:36:16.650132 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.649805    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm" event={"ID":"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25","Type":"ContainerDied","Data":"04c77138974af64140d8fe74604a9108e16d7238f75a843592a56f2247e82041"}
Nov 02 12:36:16.650132 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.649848    4464 scope.go:122] "RemoveContainer" containerID="ef10021c51e6b358bc480b49b9aa15e60f62df94bad10212eadbdeb94ea3686f"
Nov 02 12:36:16.669666 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.661194    4464 scope.go:122] "RemoveContainer" containerID="ef10021c51e6b358bc480b49b9aa15e60f62df94bad10212eadbdeb94ea3686f"
Nov 02 12:36:16.669666 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:36:16.661529    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef10021c51e6b358bc480b49b9aa15e60f62df94bad10212eadbdeb94ea3686f\": container with ID starting with ef10021c51e6b358bc480b49b9aa15e60f62df94bad10212eadbdeb94ea3686f not found: ID does not exist" containerID="ef10021c51e6b358bc480b49b9aa15e60f62df94bad10212eadbdeb94ea3686f"
Nov 02 12:36:16.669666 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.661562    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef10021c51e6b358bc480b49b9aa15e60f62df94bad10212eadbdeb94ea3686f"} err="failed to get container status \"ef10021c51e6b358bc480b49b9aa15e60f62df94bad10212eadbdeb94ea3686f\": rpc error: code = NotFound desc = could not find container \"ef10021c51e6b358bc480b49b9aa15e60f62df94bad10212eadbdeb94ea3686f\": container with ID starting with ef10021c51e6b358bc480b49b9aa15e60f62df94bad10212eadbdeb94ea3686f not found: ID does not exist"
Nov 02 12:36:16.669666 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.665514    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:36:16.737622 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.737598    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:36:16.747490 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.747470    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:36:16.766291 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.766263    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-envoy\") pod \"eb51d042-8930-45f9-89c9-fe9ad351df24\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") "
Nov 02 12:36:16.766499 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.766319    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-kube-api-access-psqxd\" (UniqueName: \"kubernetes.io/projected/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-kube-api-access-psqxd\") pod \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") "
Nov 02 12:36:16.766499 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.766356    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-data\") pod \"eb51d042-8930-45f9-89c9-fe9ad351df24\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") "
Nov 02 12:36:16.766499 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.766396    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-data\") pod \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") "
Nov 02 12:36:16.766499 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.766439    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-workload-certs\") pod \"eb51d042-8930-45f9-89c9-fe9ad351df24\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") "
Nov 02 12:36:16.766741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.766501    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istiod-ca-cert\") pod \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") "
Nov 02 12:36:16.766741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.766533    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-workload-socket\") pod \"eb51d042-8930-45f9-89c9-fe9ad351df24\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") "
Nov 02 12:36:16.766741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.766561    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-token\" (UniqueName: \"kubernetes.io/projected/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-token\") pod \"eb51d042-8930-45f9-89c9-fe9ad351df24\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") "
Nov 02 12:36:16.766741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.766595    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-credential-socket\") pod \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") "
Nov 02 12:36:16.766741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.766689    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-podinfo\") pod \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") "
Nov 02 12:36:16.766741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.766703    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-data" pod "eb51d042-8930-45f9-89c9-fe9ad351df24" (UID: "eb51d042-8930-45f9-89c9-fe9ad351df24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.766741 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.766725    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb51d042-8930-45f9-89c9-fe9ad351df24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb51d042-8930-45f9-89c9-fe9ad351df24-istiod-ca-cert\") pod \"eb51d042-8930-45f9-89c9-fe9ad351df24\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") "
Nov 02 12:36:16.767136 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.766762    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-credential-socket\") pod \"eb51d042-8930-45f9-89c9-fe9ad351df24\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") "
Nov 02 12:36:16.767136 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.766804    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-envoy\") pod \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") "
Nov 02 12:36:16.767136 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.766861    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-workload-socket\") pod \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") "
Nov 02 12:36:16.767136 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.766915    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-podinfo\") pod \"eb51d042-8930-45f9-89c9-fe9ad351df24\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") "
Nov 02 12:36:16.767136 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.766964    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb51d042-8930-45f9-89c9-fe9ad351df24-kube-api-access-tnc65\" (UniqueName: \"kubernetes.io/projected/eb51d042-8930-45f9-89c9-fe9ad351df24-kube-api-access-tnc65\") pod \"eb51d042-8930-45f9-89c9-fe9ad351df24\" (UID: \"eb51d042-8930-45f9-89c9-fe9ad351df24\") "
Nov 02 12:36:16.767136 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.767007    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-token\" (UniqueName: \"kubernetes.io/projected/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-token\") pod \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") "
Nov 02 12:36:16.767136 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.767020    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-workload-certs" pod "eb51d042-8930-45f9-89c9-fe9ad351df24" (UID: "eb51d042-8930-45f9-89c9-fe9ad351df24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.767136 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.767036    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-workload-certs\") pod \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\" (UID: \"d36f100c-b555-4ffd-b2d3-d3ffbbb69e25\") "
Nov 02 12:36:16.767593 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.767239    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-data" pod "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" (UID: "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.767593 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.767405    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-workload-socket" pod "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" (UID: "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.767593 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.767556    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.767593 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.767563    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-credential-socket" pod "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" (UID: "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.767830 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.767581    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.767830 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.767629    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.767830 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.767647    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.769448 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.769419    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istiod-ca-cert" pod "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" (UID: "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:16.770333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.769810    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-workload-certs" pod "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" (UID: "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.770333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.770083    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-envoy" pod "eb51d042-8930-45f9-89c9-fe9ad351df24" (UID: "eb51d042-8930-45f9-89c9-fe9ad351df24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.770333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.770131    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-workload-socket" pod "eb51d042-8930-45f9-89c9-fe9ad351df24" (UID: "eb51d042-8930-45f9-89c9-fe9ad351df24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.770333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.770150    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb51d042-8930-45f9-89c9-fe9ad351df24-istiod-ca-cert" pod "eb51d042-8930-45f9-89c9-fe9ad351df24" (UID: "eb51d042-8930-45f9-89c9-fe9ad351df24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:16.770639 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.770431    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-credential-socket" pod "eb51d042-8930-45f9-89c9-fe9ad351df24" (UID: "eb51d042-8930-45f9-89c9-fe9ad351df24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.770639 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.770551    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-podinfo" pod "eb51d042-8930-45f9-89c9-fe9ad351df24" (UID: "eb51d042-8930-45f9-89c9-fe9ad351df24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:36:16.770851 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.770824    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-token" pod "eb51d042-8930-45f9-89c9-fe9ad351df24" (UID: "eb51d042-8930-45f9-89c9-fe9ad351df24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:16.770851 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.770844    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-kube-api-access-psqxd" pod "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" (UID: "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25"). InnerVolumeSpecName "kube-api-access-psqxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:16.771045 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.770851    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-podinfo" pod "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" (UID: "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:36:16.772613 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.772591    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb51d042-8930-45f9-89c9-fe9ad351df24-kube-api-access-tnc65" pod "eb51d042-8930-45f9-89c9-fe9ad351df24" (UID: "eb51d042-8930-45f9-89c9-fe9ad351df24"). InnerVolumeSpecName "kube-api-access-tnc65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:16.772859 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.772835    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-envoy" pod "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" (UID: "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.772859 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.772852    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-token" pod "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" (UID: "d36f100c-b555-4ffd-b2d3-d3ffbbb69e25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:16.868422 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868390    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-podinfo\") pod \"896e3cac-30cf-492b-a20a-cec437dc6a05\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") "
Nov 02 12:36:16.868615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868439    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-workload-socket\") pod \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") "
Nov 02 12:36:16.868615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868468    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-podinfo\") pod \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") "
Nov 02 12:36:16.868615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868496    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-envoy\") pod \"896e3cac-30cf-492b-a20a-cec437dc6a05\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") "
Nov 02 12:36:16.868615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868522    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-data\") pod \"896e3cac-30cf-492b-a20a-cec437dc6a05\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") "
Nov 02 12:36:16.868615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868555    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-token\" (UniqueName: \"kubernetes.io/projected/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-token\") pod \"896e3cac-30cf-492b-a20a-cec437dc6a05\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") "
Nov 02 12:36:16.868615 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868593    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-credential-socket\") pod \"896e3cac-30cf-492b-a20a-cec437dc6a05\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") "
Nov 02 12:36:16.869006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868621    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-workload-socket\") pod \"896e3cac-30cf-492b-a20a-cec437dc6a05\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") "
Nov 02 12:36:16.869006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868641    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-workload-certs\") pod \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") "
Nov 02 12:36:16.869006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868667    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-kube-api-access-6644t\" (UniqueName: \"kubernetes.io/projected/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-kube-api-access-6644t\") pod \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") "
Nov 02 12:36:16.869006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868693    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/896e3cac-30cf-492b-a20a-cec437dc6a05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896e3cac-30cf-492b-a20a-cec437dc6a05-istiod-ca-cert\") pod \"896e3cac-30cf-492b-a20a-cec437dc6a05\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") "
Nov 02 12:36:16.869006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868711    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-token\" (UniqueName: \"kubernetes.io/projected/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-token\") pod \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") "
Nov 02 12:36:16.869006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868733    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istiod-ca-cert\") pod \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") "
Nov 02 12:36:16.869006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868749    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-credential-socket\") pod \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") "
Nov 02 12:36:16.869006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868780    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896e3cac-30cf-492b-a20a-cec437dc6a05-kube-api-access-xsgjd\" (UniqueName: \"kubernetes.io/projected/896e3cac-30cf-492b-a20a-cec437dc6a05-kube-api-access-xsgjd\") pod \"896e3cac-30cf-492b-a20a-cec437dc6a05\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") "
Nov 02 12:36:16.869006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868824    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-data\") pod \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") "
Nov 02 12:36:16.869006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868877    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-workload-certs\") pod \"896e3cac-30cf-492b-a20a-cec437dc6a05\" (UID: \"896e3cac-30cf-492b-a20a-cec437dc6a05\") "
Nov 02 12:36:16.869006 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868936    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-envoy\") pod \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\" (UID: \"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f\") "
Nov 02 12:36:16.869608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869447    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.869608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869472    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.869608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868740    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-workload-socket" pod "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" (UID: "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.869608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869488    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.869608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869523    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.869608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869545    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.869608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869561    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb51d042-8930-45f9-89c9-fe9ad351df24-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.869608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869577    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.869608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869595    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.869608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869609    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.870174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869623    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnc65\" (UniqueName: \"kubernetes.io/projected/eb51d042-8930-45f9-89c9-fe9ad351df24-kube-api-access-tnc65\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.870174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869639    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.870174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869654    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.870174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869669    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb51d042-8930-45f9-89c9-fe9ad351df24-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.870174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869683    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psqxd\" (UniqueName: \"kubernetes.io/projected/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25-kube-api-access-psqxd\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.870174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.870042    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istiod-ca-cert" pod "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" (UID: "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:16.870507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.870271    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-credential-socket" pod "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" (UID: "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.870507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.868926    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-workload-certs" pod "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" (UID: "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.870507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869018    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-credential-socket" pod "896e3cac-30cf-492b-a20a-cec437dc6a05" (UID: "896e3cac-30cf-492b-a20a-cec437dc6a05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.870507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869143    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-workload-socket" pod "896e3cac-30cf-492b-a20a-cec437dc6a05" (UID: "896e3cac-30cf-492b-a20a-cec437dc6a05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.870507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869171    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-data" pod "896e3cac-30cf-492b-a20a-cec437dc6a05" (UID: "896e3cac-30cf-492b-a20a-cec437dc6a05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.870507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.869471    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896e3cac-30cf-492b-a20a-cec437dc6a05-istiod-ca-cert" pod "896e3cac-30cf-492b-a20a-cec437dc6a05" (UID: "896e3cac-30cf-492b-a20a-cec437dc6a05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:16.870507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.870477    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-data" pod "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" (UID: "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.871217 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.871181    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-workload-certs" pod "896e3cac-30cf-492b-a20a-cec437dc6a05" (UID: "896e3cac-30cf-492b-a20a-cec437dc6a05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.871333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.871224    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-podinfo" pod "896e3cac-30cf-492b-a20a-cec437dc6a05" (UID: "896e3cac-30cf-492b-a20a-cec437dc6a05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:36:16.871333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.871232    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-envoy" pod "896e3cac-30cf-492b-a20a-cec437dc6a05" (UID: "896e3cac-30cf-492b-a20a-cec437dc6a05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.871333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.871315    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-podinfo" pod "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" (UID: "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:36:16.871540 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.871462    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-token" pod "896e3cac-30cf-492b-a20a-cec437dc6a05" (UID: "896e3cac-30cf-492b-a20a-cec437dc6a05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:16.871649 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.871624    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-kube-api-access-6644t" pod "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" (UID: "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f"). InnerVolumeSpecName "kube-api-access-6644t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:16.871755 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.871683    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-envoy" pod "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" (UID: "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:36:16.872501 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.872482    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-token" pod "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" (UID: "890b9eeb-eb6b-4cc5-beff-222fcdc94d2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:16.872914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.872889    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896e3cac-30cf-492b-a20a-cec437dc6a05-kube-api-access-xsgjd" pod "896e3cac-30cf-492b-a20a-cec437dc6a05" (UID: "896e3cac-30cf-492b-a20a-cec437dc6a05"). InnerVolumeSpecName "kube-api-access-xsgjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:16.970685 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970653    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.970685 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970682    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.970914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970697    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.970914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970710    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.970914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970724    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.970914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970739    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.970914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970763    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.970914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970776    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896e3cac-30cf-492b-a20a-cec437dc6a05-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.970914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970790    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.970914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970802    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896e3cac-30cf-492b-a20a-cec437dc6a05-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.970914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970814    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.970914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970829    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6644t\" (UniqueName: \"kubernetes.io/projected/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-kube-api-access-6644t\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.970914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970842    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896e3cac-30cf-492b-a20a-cec437dc6a05-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.970914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970857    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.970914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970872    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.970914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970890    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.970914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970900    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsgjd\" (UniqueName: \"kubernetes.io/projected/896e3cac-30cf-492b-a20a-cec437dc6a05-kube-api-access-xsgjd\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:16.970914 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:16.970911    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:17.337810 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.337775    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"]
Nov 02 12:36:17.341720 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.341697    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mpqgm"]
Nov 02 12:36:17.580929 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.580898    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" path="/var/lib/kubelet/pods/d36f100c-b555-4ffd-b2d3-d3ffbbb69e25/volumes"
Nov 02 12:36:17.657332 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.657309    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"
Nov 02 12:36:17.657734 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.657301    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv" event={"ID":"890b9eeb-eb6b-4cc5-beff-222fcdc94d2f","Type":"ContainerDied","Data":"383be9d637c7cdfc0f2e35c8ce386241d38177022e5ac5c64d075b0cc6c297e9"}
Nov 02 12:36:17.657734 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.657473    4464 scope.go:122] "RemoveContainer" containerID="f82b80bb5789034c6329a820dc78fb89a27d1b57da62eb9d50f37feb5629b1e5"
Nov 02 12:36:17.659384 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.659322    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c" event={"ID":"896e3cac-30cf-492b-a20a-cec437dc6a05","Type":"ContainerDied","Data":"fac98a43db353bccbc05f156e2e8d30c6a751686a188dff16da6a573ac48a0b8"}
Nov 02 12:36:17.659384 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.659371    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"
Nov 02 12:36:17.660954 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.660926    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4" event={"ID":"eb51d042-8930-45f9-89c9-fe9ad351df24","Type":"ContainerDied","Data":"afb4502e422822f17bf0bfd3570e5a8aef741458dad016090de0ea0824f93b4e"}
Nov 02 12:36:17.661075 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.660935    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"
Nov 02 12:36:17.668480 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.668461    4464 scope.go:122] "RemoveContainer" containerID="419f3f75d7874891e9dd2af2149c76a2ce261aee46c0ece5e5f51d184547be5a"
Nov 02 12:36:17.677138 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.677115    4464 scope.go:122] "RemoveContainer" containerID="9cf4dc362091b40edd23666bb3978ed460ed494b6e14a13c5b60cdb2eb37a3e3"
Nov 02 12:36:17.697713 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.697688    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-62g8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:36:17.697821 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.697738    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c" podUID="896e3cac-30cf-492b-a20a-cec437dc6a05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:36:17.723054 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.723021    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jthbv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:36:17.723221 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.723074    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv" podUID="890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:36:17.825950 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.825915    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"]
Nov 02 12:36:17.831111 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.831083    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-62g8c"]
Nov 02 12:36:17.946864 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.946781    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"]
Nov 02 12:36:17.952108 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:17.952083    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jthbv"]
Nov 02 12:36:18.063449 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:18.063417    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"]
Nov 02 12:36:18.069690 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:18.069664    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lwhz4"]
Nov 02 12:36:19.581805 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:19.581769    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" path="/var/lib/kubelet/pods/890b9eeb-eb6b-4cc5-beff-222fcdc94d2f/volumes"
Nov 02 12:36:19.582234 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:19.582219    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="896e3cac-30cf-492b-a20a-cec437dc6a05" path="/var/lib/kubelet/pods/896e3cac-30cf-492b-a20a-cec437dc6a05/volumes"
Nov 02 12:36:19.582612 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:19.582600    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb51d042-8930-45f9-89c9-fe9ad351df24" path="/var/lib/kubelet/pods/eb51d042-8930-45f9-89c9-fe9ad351df24/volumes"
Nov 02 12:36:28.291873 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:36:28.291838    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:36:32.118342 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:32.118302    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"]
Nov 02 12:36:32.224542 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:32.224508    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"]
Nov 02 12:36:37.119180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.119119    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd" podUID="b2b1929e-1a37-4829-aa6b-2cc7aef86f0f" containerName="payload-processing" containerID="cri-o://39a310ee72f2a2d933225a36efd41df3501616e8747299288e395bac485da8a7" gracePeriod=25
Nov 02 12:36:37.224794 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.224762    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk" podUID="6f6b2381-3780-4856-8418-f55f4ac6e082" containerName="payload-pre-processing" containerID="cri-o://fdfb5b099f0e854fb62f2d0d20572fabba89d30ffabf63010f67e9e6a06f2a5c" gracePeriod=25
Nov 02 12:36:37.369415 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.369355    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"
Nov 02 12:36:37.468171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.468147    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2b1929e-1a37-4829-aa6b-2cc7aef86f0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2b1929e-1a37-4829-aa6b-2cc7aef86f0f-plugins-config-volume\") pod \"b2b1929e-1a37-4829-aa6b-2cc7aef86f0f\" (UID: \"b2b1929e-1a37-4829-aa6b-2cc7aef86f0f\") "
Nov 02 12:36:37.468305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.468206    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b1929e-1a37-4829-aa6b-2cc7aef86f0f-kube-api-access-rfcwq\" (UniqueName: \"kubernetes.io/projected/b2b1929e-1a37-4829-aa6b-2cc7aef86f0f-kube-api-access-rfcwq\") pod \"b2b1929e-1a37-4829-aa6b-2cc7aef86f0f\" (UID: \"b2b1929e-1a37-4829-aa6b-2cc7aef86f0f\") "
Nov 02 12:36:37.468516 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.468492    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b1929e-1a37-4829-aa6b-2cc7aef86f0f-plugins-config-volume" pod "b2b1929e-1a37-4829-aa6b-2cc7aef86f0f" (UID: "b2b1929e-1a37-4829-aa6b-2cc7aef86f0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:37.470269 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.470245    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b1929e-1a37-4829-aa6b-2cc7aef86f0f-kube-api-access-rfcwq" pod "b2b1929e-1a37-4829-aa6b-2cc7aef86f0f" (UID: "b2b1929e-1a37-4829-aa6b-2cc7aef86f0f"). InnerVolumeSpecName "kube-api-access-rfcwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:37.475788 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.475767    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"
Nov 02 12:36:37.569103 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.569075    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f6b2381-3780-4856-8418-f55f4ac6e082-kube-api-access-997gg\" (UniqueName: \"kubernetes.io/projected/6f6b2381-3780-4856-8418-f55f4ac6e082-kube-api-access-997gg\") pod \"6f6b2381-3780-4856-8418-f55f4ac6e082\" (UID: \"6f6b2381-3780-4856-8418-f55f4ac6e082\") "
Nov 02 12:36:37.569280 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.569229    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f6b2381-3780-4856-8418-f55f4ac6e082-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f6b2381-3780-4856-8418-f55f4ac6e082-plugins-config-volume\") pod \"6f6b2381-3780-4856-8418-f55f4ac6e082\" (UID: \"6f6b2381-3780-4856-8418-f55f4ac6e082\") "
Nov 02 12:36:37.569426 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.569414    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2b1929e-1a37-4829-aa6b-2cc7aef86f0f-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:37.569497 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.569428    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfcwq\" (UniqueName: \"kubernetes.io/projected/b2b1929e-1a37-4829-aa6b-2cc7aef86f0f-kube-api-access-rfcwq\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:37.569571 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.569550    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f6b2381-3780-4856-8418-f55f4ac6e082-plugins-config-volume" pod "6f6b2381-3780-4856-8418-f55f4ac6e082" (UID: "6f6b2381-3780-4856-8418-f55f4ac6e082"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:37.571120 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.571100    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6b2381-3780-4856-8418-f55f4ac6e082-kube-api-access-997gg" pod "6f6b2381-3780-4856-8418-f55f4ac6e082" (UID: "6f6b2381-3780-4856-8418-f55f4ac6e082"). InnerVolumeSpecName "kube-api-access-997gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:37.670094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.670032    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f6b2381-3780-4856-8418-f55f4ac6e082-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:37.670094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.670056    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-997gg\" (UniqueName: \"kubernetes.io/projected/6f6b2381-3780-4856-8418-f55f4ac6e082-kube-api-access-997gg\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:36:37.773657 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.773625    4464 generic.go:356] "Generic (PLEG): container finished" podID="b2b1929e-1a37-4829-aa6b-2cc7aef86f0f" containerID="39a310ee72f2a2d933225a36efd41df3501616e8747299288e395bac485da8a7" exitCode=0
Nov 02 12:36:37.773813 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.773666    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"
Nov 02 12:36:37.773813 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.773711    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd" event={"ID":"b2b1929e-1a37-4829-aa6b-2cc7aef86f0f","Type":"ContainerDied","Data":"39a310ee72f2a2d933225a36efd41df3501616e8747299288e395bac485da8a7"}
Nov 02 12:36:37.773813 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.773765    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd" event={"ID":"b2b1929e-1a37-4829-aa6b-2cc7aef86f0f","Type":"ContainerDied","Data":"fde590a5a5158d3af0ab6c2eca6adea202c59f5198b4d00b43447c8a67d08197"}
Nov 02 12:36:37.773813 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.773792    4464 scope.go:122] "RemoveContainer" containerID="39a310ee72f2a2d933225a36efd41df3501616e8747299288e395bac485da8a7"
Nov 02 12:36:37.775452 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.775428    4464 generic.go:356] "Generic (PLEG): container finished" podID="6f6b2381-3780-4856-8418-f55f4ac6e082" containerID="fdfb5b099f0e854fb62f2d0d20572fabba89d30ffabf63010f67e9e6a06f2a5c" exitCode=0
Nov 02 12:36:37.775550 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.775467    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"
Nov 02 12:36:37.775550 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.775480    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk" event={"ID":"6f6b2381-3780-4856-8418-f55f4ac6e082","Type":"ContainerDied","Data":"fdfb5b099f0e854fb62f2d0d20572fabba89d30ffabf63010f67e9e6a06f2a5c"}
Nov 02 12:36:37.775550 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.775516    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk" event={"ID":"6f6b2381-3780-4856-8418-f55f4ac6e082","Type":"ContainerDied","Data":"e23c0d9cc47d2f5657febe50850b8717108897ae90f80bb40c9fbb01df9d57ec"}
Nov 02 12:36:37.783104 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.783083    4464 scope.go:122] "RemoveContainer" containerID="39a310ee72f2a2d933225a36efd41df3501616e8747299288e395bac485da8a7"
Nov 02 12:36:37.783379 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:36:37.783356    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a310ee72f2a2d933225a36efd41df3501616e8747299288e395bac485da8a7\": container with ID starting with 39a310ee72f2a2d933225a36efd41df3501616e8747299288e395bac485da8a7 not found: ID does not exist" containerID="39a310ee72f2a2d933225a36efd41df3501616e8747299288e395bac485da8a7"
Nov 02 12:36:37.783481 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.783385    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a310ee72f2a2d933225a36efd41df3501616e8747299288e395bac485da8a7"} err="failed to get container status \"39a310ee72f2a2d933225a36efd41df3501616e8747299288e395bac485da8a7\": rpc error: code = NotFound desc = could not find container \"39a310ee72f2a2d933225a36efd41df3501616e8747299288e395bac485da8a7\": container with ID starting with 39a310ee72f2a2d933225a36efd41df3501616e8747299288e395bac485da8a7 not found: ID does not exist"
Nov 02 12:36:37.783481 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.783402    4464 scope.go:122] "RemoveContainer" containerID="fdfb5b099f0e854fb62f2d0d20572fabba89d30ffabf63010f67e9e6a06f2a5c"
Nov 02 12:36:37.791235 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.791216    4464 scope.go:122] "RemoveContainer" containerID="fdfb5b099f0e854fb62f2d0d20572fabba89d30ffabf63010f67e9e6a06f2a5c"
Nov 02 12:36:37.791473 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:36:37.791455    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdfb5b099f0e854fb62f2d0d20572fabba89d30ffabf63010f67e9e6a06f2a5c\": container with ID starting with fdfb5b099f0e854fb62f2d0d20572fabba89d30ffabf63010f67e9e6a06f2a5c not found: ID does not exist" containerID="fdfb5b099f0e854fb62f2d0d20572fabba89d30ffabf63010f67e9e6a06f2a5c"
Nov 02 12:36:37.791521 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.791480    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdfb5b099f0e854fb62f2d0d20572fabba89d30ffabf63010f67e9e6a06f2a5c"} err="failed to get container status \"fdfb5b099f0e854fb62f2d0d20572fabba89d30ffabf63010f67e9e6a06f2a5c\": rpc error: code = NotFound desc = could not find container \"fdfb5b099f0e854fb62f2d0d20572fabba89d30ffabf63010f67e9e6a06f2a5c\": container with ID starting with fdfb5b099f0e854fb62f2d0d20572fabba89d30ffabf63010f67e9e6a06f2a5c not found: ID does not exist"
Nov 02 12:36:37.940073 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.939999    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"]
Nov 02 12:36:37.944223 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:37.944197    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-6373b9-85f9c445d5-x85bd"]
Nov 02 12:36:38.054750 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:38.054712    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"]
Nov 02 12:36:38.061147 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:38.061124    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-6373b9-5c9bfff747-cbblk"]
Nov 02 12:36:39.581793 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:39.581758    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f6b2381-3780-4856-8418-f55f4ac6e082" path="/var/lib/kubelet/pods/6f6b2381-3780-4856-8418-f55f4ac6e082/volumes"
Nov 02 12:36:39.582196 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:39.582159    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2b1929e-1a37-4829-aa6b-2cc7aef86f0f" path="/var/lib/kubelet/pods/b2b1929e-1a37-4829-aa6b-2cc7aef86f0f/volumes"
Nov 02 12:36:51.292957 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:51.292919    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:36:55.637415 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637384    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"]
Nov 02 12:36:55.637846 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637833    4464 cpu_manager.go:414] "containerMap: removing container" podUID="51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" containerName="istio-proxy"
Nov 02 12:36:55.637846 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637844    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" containerName="istio-proxy"
Nov 02 12:36:55.637967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637860    4464 cpu_manager.go:414] "containerMap: removing container" podUID="d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" containerName="istio-proxy"
Nov 02 12:36:55.637967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637866    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" containerName="istio-proxy"
Nov 02 12:36:55.637967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637876    4464 cpu_manager.go:414] "containerMap: removing container" podUID="666a9bf6-5863-44ce-9574-26ca5ee9b856" containerName="istio-proxy"
Nov 02 12:36:55.637967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637881    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="666a9bf6-5863-44ce-9574-26ca5ee9b856" containerName="istio-proxy"
Nov 02 12:36:55.637967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637888    4464 cpu_manager.go:414] "containerMap: removing container" podUID="890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" containerName="istio-proxy"
Nov 02 12:36:55.637967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637894    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" containerName="istio-proxy"
Nov 02 12:36:55.637967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637900    4464 cpu_manager.go:414] "containerMap: removing container" podUID="51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" containerName="istio-proxy"
Nov 02 12:36:55.637967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637904    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" containerName="istio-proxy"
Nov 02 12:36:55.637967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637911    4464 cpu_manager.go:414] "containerMap: removing container" podUID="b2b1929e-1a37-4829-aa6b-2cc7aef86f0f" containerName="payload-processing"
Nov 02 12:36:55.637967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637916    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b1929e-1a37-4829-aa6b-2cc7aef86f0f" containerName="payload-processing"
Nov 02 12:36:55.637967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637927    4464 cpu_manager.go:414] "containerMap: removing container" podUID="09dda7c2-acb9-43ef-a3c2-e7147de8d3be" containerName="istio-proxy"
Nov 02 12:36:55.637967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637931    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09dda7c2-acb9-43ef-a3c2-e7147de8d3be" containerName="istio-proxy"
Nov 02 12:36:55.637967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637941    4464 cpu_manager.go:414] "containerMap: removing container" podUID="c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5" containerName="payload-processing"
Nov 02 12:36:55.637967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637945    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5" containerName="payload-processing"
Nov 02 12:36:55.637967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637959    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6f6b2381-3780-4856-8418-f55f4ac6e082" containerName="payload-pre-processing"
Nov 02 12:36:55.637967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637966    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f6b2381-3780-4856-8418-f55f4ac6e082" containerName="payload-pre-processing"
Nov 02 12:36:55.638952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637993    4464 cpu_manager.go:414] "containerMap: removing container" podUID="896e3cac-30cf-492b-a20a-cec437dc6a05" containerName="istio-proxy"
Nov 02 12:36:55.638952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.637998    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="896e3cac-30cf-492b-a20a-cec437dc6a05" containerName="istio-proxy"
Nov 02 12:36:55.638952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.638013    4464 cpu_manager.go:414] "containerMap: removing container" podUID="eb51d042-8930-45f9-89c9-fe9ad351df24" containerName="istio-proxy"
Nov 02 12:36:55.638952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.638018    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb51d042-8930-45f9-89c9-fe9ad351df24" containerName="istio-proxy"
Nov 02 12:36:55.638952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.638028    4464 cpu_manager.go:414] "containerMap: removing container" podUID="fcfaa90b-9297-4cfe-b835-29d2454322ac" containerName="payload-pre-processing"
Nov 02 12:36:55.638952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.638032    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcfaa90b-9297-4cfe-b835-29d2454322ac" containerName="payload-pre-processing"
Nov 02 12:36:55.638952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.638099    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb51d042-8930-45f9-89c9-fe9ad351df24" containerName="istio-proxy"
Nov 02 12:36:55.638952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.638110    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcfaa90b-9297-4cfe-b835-29d2454322ac" containerName="payload-pre-processing"
Nov 02 12:36:55.638952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.638119    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" containerName="istio-proxy"
Nov 02 12:36:55.638952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.638128    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="d36f100c-b555-4ffd-b2d3-d3ffbbb69e25" containerName="istio-proxy"
Nov 02 12:36:55.638952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.638136    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="666a9bf6-5863-44ce-9574-26ca5ee9b856" containerName="istio-proxy"
Nov 02 12:36:55.638952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.638143    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="890b9eeb-eb6b-4cc5-beff-222fcdc94d2f" containerName="istio-proxy"
Nov 02 12:36:55.638952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.638147    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b1929e-1a37-4829-aa6b-2cc7aef86f0f" containerName="payload-processing"
Nov 02 12:36:55.638952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.638153    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="09dda7c2-acb9-43ef-a3c2-e7147de8d3be" containerName="istio-proxy"
Nov 02 12:36:55.638952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.638158    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56fd4ed-661a-4a95-b6cb-062ff2aa2ca5" containerName="payload-processing"
Nov 02 12:36:55.638952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.638164    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f6b2381-3780-4856-8418-f55f4ac6e082" containerName="payload-pre-processing"
Nov 02 12:36:55.638952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.638170    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="896e3cac-30cf-492b-a20a-cec437dc6a05" containerName="istio-proxy"
Nov 02 12:36:55.641536 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.641520    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.695612 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.695576    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-02c1d2-openshift-default-dockercfg-65rml\""
Nov 02 12:36:55.748893 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.748857    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"]
Nov 02 12:36:55.751338 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.751312    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-podinfo\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.751486 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.751348    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-envoy\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.751486 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.751384    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-credential-socket\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.751586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.751502    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-token\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.751586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.751559    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-data\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.751696 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.751651    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zb4r\" (UniqueName: \"kubernetes.io/projected/745a1de1-9414-4a13-bacd-c13d0a0ce101-kube-api-access-2zb4r\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.751757 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.751738    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-workload-socket\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.751815 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.751768    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745a1de1-9414-4a13-bacd-c13d0a0ce101-istiod-ca-cert\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.751892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.751876    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-workload-certs\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.852510 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.852476    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-workload-certs\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.852704 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.852550    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-podinfo\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.852704 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.852576    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-envoy\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.852704 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.852609    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-credential-socket\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.852704 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.852644    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-token\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.852704 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.852700    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-data\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.852969 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.852738    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zb4r\" (UniqueName: \"kubernetes.io/projected/745a1de1-9414-4a13-bacd-c13d0a0ce101-kube-api-access-2zb4r\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.852969 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.852899    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-workload-socket\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.852969 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.852946    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745a1de1-9414-4a13-bacd-c13d0a0ce101-istiod-ca-cert\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.852969 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.852953    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-workload-certs\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.853228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.853011    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-credential-socket\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.853481 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.853357    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-workload-socket\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.853481 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.853416    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-data\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.853663 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.853604    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745a1de1-9414-4a13-bacd-c13d0a0ce101-istiod-ca-cert\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.855130 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.855108    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-envoy\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.855322 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.855304    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-podinfo\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.914718 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.914658    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-token\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.914818 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.914742    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zb4r\" (UniqueName: \"kubernetes.io/projected/745a1de1-9414-4a13-bacd-c13d0a0ce101-kube-api-access-2zb4r\") pod \"e2e-noten-02c1d2-openshift-default-84ff487789-gczvd\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") " pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:55.955597 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:55.955574    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:56.384067 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:56.384038    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"]
Nov 02 12:36:56.385160 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:36:56.385137    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745a1de1_9414_4a13_bacd_c13d0a0ce101.slice/crio-3e3ba764a84e3edfa215d27c386fc95a13799514a22f5534672c1f3b62b7b4e6 WatchSource:0}: Error finding container 3e3ba764a84e3edfa215d27c386fc95a13799514a22f5534672c1f3b62b7b4e6: Status 404 returned error can't find the container with id 3e3ba764a84e3edfa215d27c386fc95a13799514a22f5534672c1f3b62b7b4e6
Nov 02 12:36:56.387373 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:56.387342    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:36:56.387446 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:56.387410    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:36:56.387446 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:56.387439    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 12:36:56.902164 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:56.902127    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd" event={"ID":"745a1de1-9414-4a13-bacd-c13d0a0ce101","Type":"ContainerStarted","Data":"6910d270b59e01fcce7cae9fe5617b0aca9117b2fc9dc7012d65cc0e120e90c2"}
Nov 02 12:36:56.902651 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:56.902177    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd" event={"ID":"745a1de1-9414-4a13-bacd-c13d0a0ce101","Type":"ContainerStarted","Data":"3e3ba764a84e3edfa215d27c386fc95a13799514a22f5534672c1f3b62b7b4e6"}
Nov 02 12:36:56.956088 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:56.956060    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:57.026172 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:57.026128    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd" podStartSLOduration=2.026115634 podStartE2EDuration="2.026115634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:57.024143414 +0000 UTC m=+2445.587851825" watchObservedRunningTime="2026-11-02 12:36:57.026115634 +0000 UTC m=+2445.589824104"
Nov 02 12:36:57.965051 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:57.965018    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:58.290755 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:36:58.290721    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:36:58.913328 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:58.913294    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:36:58.914339 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:36:58.914321    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:37:13.722478 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:13.722443    4464 scope.go:122] "RemoveContainer" containerID="f2380cb4c920f12cbf1ece26523f00eb759eac352b4ca42d320b74c8cd0216c2"
Nov 02 12:37:13.730993 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:13.730959    4464 scope.go:122] "RemoveContainer" containerID="75d0bd1b036e1be961e5e52f9eabdb94a9221d26b843d8757935567f7b238c92"
Nov 02 12:37:13.738687 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:13.738671    4464 scope.go:122] "RemoveContainer" containerID="07d2b813b2d4b743a5cc5e6393043b00bbf4e5c9685314a8f4f96c084af52516"
Nov 02 12:37:22.023372 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:22.023333    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"]
Nov 02 12:37:22.025715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:22.023601    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd" podUID="745a1de1-9414-4a13-bacd-c13d0a0ce101" containerName="istio-proxy" containerID="cri-o://6910d270b59e01fcce7cae9fe5617b0aca9117b2fc9dc7012d65cc0e120e90c2" gracePeriod=30
Nov 02 12:37:27.258545 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.258520    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:37:27.354065 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.353963    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-workload-socket\") pod \"745a1de1-9414-4a13-bacd-c13d0a0ce101\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") "
Nov 02 12:37:27.354239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.354069    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-credential-socket\") pod \"745a1de1-9414-4a13-bacd-c13d0a0ce101\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") "
Nov 02 12:37:27.354239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.354090    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-workload-certs\") pod \"745a1de1-9414-4a13-bacd-c13d0a0ce101\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") "
Nov 02 12:37:27.354239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.354121    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-envoy\") pod \"745a1de1-9414-4a13-bacd-c13d0a0ce101\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") "
Nov 02 12:37:27.354239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.354159    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-token\" (UniqueName: \"kubernetes.io/projected/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-token\") pod \"745a1de1-9414-4a13-bacd-c13d0a0ce101\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") "
Nov 02 12:37:27.354239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.354198    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745a1de1-9414-4a13-bacd-c13d0a0ce101-kube-api-access-2zb4r\" (UniqueName: \"kubernetes.io/projected/745a1de1-9414-4a13-bacd-c13d0a0ce101-kube-api-access-2zb4r\") pod \"745a1de1-9414-4a13-bacd-c13d0a0ce101\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") "
Nov 02 12:37:27.354239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.354202    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-workload-socket" pod "745a1de1-9414-4a13-bacd-c13d0a0ce101" (UID: "745a1de1-9414-4a13-bacd-c13d0a0ce101"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:37:27.354586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.354252    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/745a1de1-9414-4a13-bacd-c13d0a0ce101-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745a1de1-9414-4a13-bacd-c13d0a0ce101-istiod-ca-cert\") pod \"745a1de1-9414-4a13-bacd-c13d0a0ce101\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") "
Nov 02 12:37:27.354586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.354403    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-credential-socket" pod "745a1de1-9414-4a13-bacd-c13d0a0ce101" (UID: "745a1de1-9414-4a13-bacd-c13d0a0ce101"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:37:27.354586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.354469    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-podinfo\") pod \"745a1de1-9414-4a13-bacd-c13d0a0ce101\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") "
Nov 02 12:37:27.354586 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.354502    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-data\") pod \"745a1de1-9414-4a13-bacd-c13d0a0ce101\" (UID: \"745a1de1-9414-4a13-bacd-c13d0a0ce101\") "
Nov 02 12:37:27.354852 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.354617    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745a1de1-9414-4a13-bacd-c13d0a0ce101-istiod-ca-cert" pod "745a1de1-9414-4a13-bacd-c13d0a0ce101" (UID: "745a1de1-9414-4a13-bacd-c13d0a0ce101"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:37:27.354906 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.354863    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-data" pod "745a1de1-9414-4a13-bacd-c13d0a0ce101" (UID: "745a1de1-9414-4a13-bacd-c13d0a0ce101"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:37:27.355116 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.354961    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:37:27.355116 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.355015    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745a1de1-9414-4a13-bacd-c13d0a0ce101-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:37:27.355116 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.355029    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:37:27.355116 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.355043    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:37:27.355116 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.355070    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-workload-certs" pod "745a1de1-9414-4a13-bacd-c13d0a0ce101" (UID: "745a1de1-9414-4a13-bacd-c13d0a0ce101"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:37:27.356613 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.356584    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-token" pod "745a1de1-9414-4a13-bacd-c13d0a0ce101" (UID: "745a1de1-9414-4a13-bacd-c13d0a0ce101"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:37:27.356613 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.356609    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-podinfo" pod "745a1de1-9414-4a13-bacd-c13d0a0ce101" (UID: "745a1de1-9414-4a13-bacd-c13d0a0ce101"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:37:27.356787 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.356742    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-envoy" pod "745a1de1-9414-4a13-bacd-c13d0a0ce101" (UID: "745a1de1-9414-4a13-bacd-c13d0a0ce101"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:37:27.356836 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.356803    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745a1de1-9414-4a13-bacd-c13d0a0ce101-kube-api-access-2zb4r" pod "745a1de1-9414-4a13-bacd-c13d0a0ce101" (UID: "745a1de1-9414-4a13-bacd-c13d0a0ce101"). InnerVolumeSpecName "kube-api-access-2zb4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:37:27.455641 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.455599    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zb4r\" (UniqueName: \"kubernetes.io/projected/745a1de1-9414-4a13-bacd-c13d0a0ce101-kube-api-access-2zb4r\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:37:27.455641 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.455638    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:37:27.455641 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.455649    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:37:27.455641 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.455658    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:37:27.455924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:27.455667    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745a1de1-9414-4a13-bacd-c13d0a0ce101-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:37:28.066870 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:28.066835    4464 generic.go:356] "Generic (PLEG): container finished" podID="745a1de1-9414-4a13-bacd-c13d0a0ce101" containerID="6910d270b59e01fcce7cae9fe5617b0aca9117b2fc9dc7012d65cc0e120e90c2" exitCode=0
Nov 02 12:37:28.067061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:28.066898    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"
Nov 02 12:37:28.067061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:28.066899    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd" event={"ID":"745a1de1-9414-4a13-bacd-c13d0a0ce101","Type":"ContainerDied","Data":"6910d270b59e01fcce7cae9fe5617b0aca9117b2fc9dc7012d65cc0e120e90c2"}
Nov 02 12:37:28.067061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:28.067024    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd" event={"ID":"745a1de1-9414-4a13-bacd-c13d0a0ce101","Type":"ContainerDied","Data":"3e3ba764a84e3edfa215d27c386fc95a13799514a22f5534672c1f3b62b7b4e6"}
Nov 02 12:37:28.067061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:28.067044    4464 scope.go:122] "RemoveContainer" containerID="6910d270b59e01fcce7cae9fe5617b0aca9117b2fc9dc7012d65cc0e120e90c2"
Nov 02 12:37:28.076223 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:28.076203    4464 scope.go:122] "RemoveContainer" containerID="6910d270b59e01fcce7cae9fe5617b0aca9117b2fc9dc7012d65cc0e120e90c2"
Nov 02 12:37:28.076483 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:37:28.076466    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6910d270b59e01fcce7cae9fe5617b0aca9117b2fc9dc7012d65cc0e120e90c2\": container with ID starting with 6910d270b59e01fcce7cae9fe5617b0aca9117b2fc9dc7012d65cc0e120e90c2 not found: ID does not exist" containerID="6910d270b59e01fcce7cae9fe5617b0aca9117b2fc9dc7012d65cc0e120e90c2"
Nov 02 12:37:28.076524 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:28.076492    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6910d270b59e01fcce7cae9fe5617b0aca9117b2fc9dc7012d65cc0e120e90c2"} err="failed to get container status \"6910d270b59e01fcce7cae9fe5617b0aca9117b2fc9dc7012d65cc0e120e90c2\": rpc error: code = NotFound desc = could not find container \"6910d270b59e01fcce7cae9fe5617b0aca9117b2fc9dc7012d65cc0e120e90c2\": container with ID starting with 6910d270b59e01fcce7cae9fe5617b0aca9117b2fc9dc7012d65cc0e120e90c2 not found: ID does not exist"
Nov 02 12:37:28.286311 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:28.286281    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"]
Nov 02 12:37:28.292029 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:37:28.292000    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:37:28.337536 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:28.337475    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-02c1d2-openshift-default-84ff487789-gczvd"]
Nov 02 12:37:29.580720 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:37:29.580690    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="745a1de1-9414-4a13-bacd-c13d0a0ce101" path="/var/lib/kubelet/pods/745a1de1-9414-4a13-bacd-c13d0a0ce101/volumes"
Nov 02 12:37:58.291154 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:37:58.291119    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:38:01.048358 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:01.048295    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"]
Nov 02 12:38:01.098614 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:01.098579    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa771475-hclwn"]
Nov 02 12:38:01.580704 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:01.580670    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ec1103-8501-449e-9892-2a1cd83aa9b0" path="/var/lib/kubelet/pods/17ec1103-8501-449e-9892-2a1cd83aa9b0/volumes"
Nov 02 12:38:10.182758 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:10.182716    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"]
Nov 02 12:38:13.980753 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:13.980718    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"]
Nov 02 12:38:13.981199 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:13.981186    4464 cpu_manager.go:414] "containerMap: removing container" podUID="745a1de1-9414-4a13-bacd-c13d0a0ce101" containerName="istio-proxy"
Nov 02 12:38:13.981261 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:13.981201    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="745a1de1-9414-4a13-bacd-c13d0a0ce101" containerName="istio-proxy"
Nov 02 12:38:13.981300 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:13.981287    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d041eb-5ae8-46ea-b1cc-cebfa1973a1e" containerName="istio-proxy"
Nov 02 12:38:13.981345 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:13.981303    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="745a1de1-9414-4a13-bacd-c13d0a0ce101" containerName="istio-proxy"
Nov 02 12:38:13.985153 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:13.985136    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"
Nov 02 12:38:14.040336 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.040304    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"routing-overlay\""
Nov 02 12:38:14.040533 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.040412    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-14e0f93131b7\""
Nov 02 12:38:14.040533 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.040434    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"openshift-service-ca.crt\""
Nov 02 12:38:14.091084 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.090993    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3a78b4\""
Nov 02 12:38:14.091287 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.091113    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-4b8a830d91a3\""
Nov 02 12:38:14.091287 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.091158    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-50db61f22b2e\""
Nov 02 12:38:14.091287 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.091164    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"kube-root-ca.crt\""
Nov 02 12:38:14.091287 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.091262    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"payload-processing-external-model-e2e--a8ab13a3-dockercfg-wmcbx\""
Nov 02 12:38:14.093361 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.093338    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"]
Nov 02 12:38:14.106840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.106816    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqlgt\" (UniqueName: \"kubernetes.io/projected/27d181d3-3802-44f7-ace0-e66f40244776-kube-api-access-nqlgt\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67\" (UID: \"27d181d3-3802-44f7-ace0-e66f40244776\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"
Nov 02 12:38:14.106957 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.106849    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27d181d3-3802-44f7-ace0-e66f40244776-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67\" (UID: \"27d181d3-3802-44f7-ace0-e66f40244776\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"
Nov 02 12:38:14.106957 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.106880    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d181d3-3802-44f7-ace0-e66f40244776-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67\" (UID: \"27d181d3-3802-44f7-ace0-e66f40244776\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"
Nov 02 12:38:14.107095 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.107005    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27d181d3-3802-44f7-ace0-e66f40244776-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67\" (UID: \"27d181d3-3802-44f7-ace0-e66f40244776\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"
Nov 02 12:38:14.143884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.143853    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"]
Nov 02 12:38:14.145770 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.145752    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9aabe51-dpt82"]
Nov 02 12:38:14.208241 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.208207    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqlgt\" (UniqueName: \"kubernetes.io/projected/27d181d3-3802-44f7-ace0-e66f40244776-kube-api-access-nqlgt\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67\" (UID: \"27d181d3-3802-44f7-ace0-e66f40244776\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"
Nov 02 12:38:14.208417 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.208248    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27d181d3-3802-44f7-ace0-e66f40244776-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67\" (UID: \"27d181d3-3802-44f7-ace0-e66f40244776\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"
Nov 02 12:38:14.208417 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.208281    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d181d3-3802-44f7-ace0-e66f40244776-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67\" (UID: \"27d181d3-3802-44f7-ace0-e66f40244776\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"
Nov 02 12:38:14.208417 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.208316    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27d181d3-3802-44f7-ace0-e66f40244776-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67\" (UID: \"27d181d3-3802-44f7-ace0-e66f40244776\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"
Nov 02 12:38:14.209065 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.209042    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27d181d3-3802-44f7-ace0-e66f40244776-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67\" (UID: \"27d181d3-3802-44f7-ace0-e66f40244776\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"
Nov 02 12:38:14.209170 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.209105    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d181d3-3802-44f7-ace0-e66f40244776-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67\" (UID: \"27d181d3-3802-44f7-ace0-e66f40244776\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"
Nov 02 12:38:14.211036 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.211017    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27d181d3-3802-44f7-ace0-e66f40244776-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67\" (UID: \"27d181d3-3802-44f7-ace0-e66f40244776\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"
Nov 02 12:38:14.269871 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.269846    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqlgt\" (UniqueName: \"kubernetes.io/projected/27d181d3-3802-44f7-ace0-e66f40244776-kube-api-access-nqlgt\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67\" (UID: \"27d181d3-3802-44f7-ace0-e66f40244776\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"
Nov 02 12:38:14.297735 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.297706    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"
Nov 02 12:38:14.671082 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:14.671056    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"]
Nov 02 12:38:14.672962 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:38:14.672935    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d181d3_3802_44f7_ace0_e66f40244776.slice/crio-abc7c7e762077026afb567136b7fb0019d3c65f172abde7dacd5e7164eb6b8c0 WatchSource:0}: Error finding container abc7c7e762077026afb567136b7fb0019d3c65f172abde7dacd5e7164eb6b8c0: Status 404 returned error can't find the container with id abc7c7e762077026afb567136b7fb0019d3c65f172abde7dacd5e7164eb6b8c0
Nov 02 12:38:15.183751 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:15.183711    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r" podUID="fc81545d-7d40-4b73-9e60-69d2348cfddd" containerName="payload-processing" containerID="cri-o://afb4c368ae4bf0971015cca0c47994b11707388ec7b58fa35d6c8b5f746c2bbd" gracePeriod=25
Nov 02 12:38:15.316429 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:15.316390    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67" event={"ID":"27d181d3-3802-44f7-ace0-e66f40244776","Type":"ContainerStarted","Data":"c29ca4587fe88f78d2eed57a99c6d349d91e1b55cd9735d95ae57f165c4e38d6"}
Nov 02 12:38:15.316593 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:15.316449    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67" event={"ID":"27d181d3-3802-44f7-ace0-e66f40244776","Type":"ContainerStarted","Data":"abc7c7e762077026afb567136b7fb0019d3c65f172abde7dacd5e7164eb6b8c0"}
Nov 02 12:38:15.316593 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:15.316472    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"
Nov 02 12:38:15.435438 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:15.435350    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67" podStartSLOduration=2.435337314 podStartE2EDuration="2.435337314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:15.43439929 +0000 UTC m=+2523.998107702" watchObservedRunningTime="2026-11-02 12:38:15.435337314 +0000 UTC m=+2523.999045723"
Nov 02 12:38:15.587866 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:15.587831    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb7ab1ff-866c-4d73-bccb-7eded744c791" path="/var/lib/kubelet/pods/bb7ab1ff-866c-4d73-bccb-7eded744c791/volumes"
Nov 02 12:38:16.562218 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:16.562185    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:38:18.815406 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:18.815372    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"]
Nov 02 12:38:18.820217 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:18.820201    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"
Nov 02 12:38:18.874600 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:18.874570    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:38:18.925493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:18.925452    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:38:18.925493 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:18.925476    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:38:18.925685 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:18.925572    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:38:18.927630 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:18.927610    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"]
Nov 02 12:38:18.963072 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:18.963043    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b09daaa3-1e6f-4bb0-9437-6d440e085446-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77cea573-kk5ln\" (UID: \"b09daaa3-1e6f-4bb0-9437-6d440e085446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"
Nov 02 12:38:18.963194 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:18.963106    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8twhm\" (UniqueName: \"kubernetes.io/projected/b09daaa3-1e6f-4bb0-9437-6d440e085446-kube-api-access-8twhm\") pod \"maas-api-revoke-sub-77cea573-kk5ln\" (UID: \"b09daaa3-1e6f-4bb0-9437-6d440e085446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"
Nov 02 12:38:18.963261 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:18.963219    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b09daaa3-1e6f-4bb0-9437-6d440e085446-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77cea573-kk5ln\" (UID: \"b09daaa3-1e6f-4bb0-9437-6d440e085446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"
Nov 02 12:38:19.064719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:19.064690    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b09daaa3-1e6f-4bb0-9437-6d440e085446-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77cea573-kk5ln\" (UID: \"b09daaa3-1e6f-4bb0-9437-6d440e085446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"
Nov 02 12:38:19.064855 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:19.064798    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b09daaa3-1e6f-4bb0-9437-6d440e085446-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77cea573-kk5ln\" (UID: \"b09daaa3-1e6f-4bb0-9437-6d440e085446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"
Nov 02 12:38:19.064855 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:19.064832    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8twhm\" (UniqueName: \"kubernetes.io/projected/b09daaa3-1e6f-4bb0-9437-6d440e085446-kube-api-access-8twhm\") pod \"maas-api-revoke-sub-77cea573-kk5ln\" (UID: \"b09daaa3-1e6f-4bb0-9437-6d440e085446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"
Nov 02 12:38:19.065470 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:19.065427    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b09daaa3-1e6f-4bb0-9437-6d440e085446-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77cea573-kk5ln\" (UID: \"b09daaa3-1e6f-4bb0-9437-6d440e085446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"
Nov 02 12:38:19.067002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:19.066971    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b09daaa3-1e6f-4bb0-9437-6d440e085446-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77cea573-kk5ln\" (UID: \"b09daaa3-1e6f-4bb0-9437-6d440e085446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"
Nov 02 12:38:19.126622 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:19.126591    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8twhm\" (UniqueName: \"kubernetes.io/projected/b09daaa3-1e6f-4bb0-9437-6d440e085446-kube-api-access-8twhm\") pod \"maas-api-revoke-sub-77cea573-kk5ln\" (UID: \"b09daaa3-1e6f-4bb0-9437-6d440e085446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"
Nov 02 12:38:19.133412 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:19.133394    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"
Nov 02 12:38:19.223319 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:19.223288    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r" podUID="fc81545d-7d40-4b73-9e60-69d2348cfddd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:38:19.513436 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:19.511251    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"]
Nov 02 12:38:19.515202 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:38:19.515173    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09daaa3_1e6f_4bb0_9437_6d440e085446.slice/crio-384a7094b6e3235f215ab153522a7b987df5e673e99801faccdd71c28ed55a50 WatchSource:0}: Error finding container 384a7094b6e3235f215ab153522a7b987df5e673e99801faccdd71c28ed55a50: Status 404 returned error can't find the container with id 384a7094b6e3235f215ab153522a7b987df5e673e99801faccdd71c28ed55a50
Nov 02 12:38:19.525546 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:19.525519    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"]
Nov 02 12:38:19.525774 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:19.525749    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb" podUID="4b684afa-613a-43b5-b10d-0f0ece8da04c" containerName="istio-proxy" containerID="cri-o://da2129fb600ab49a1312806990197fed9e3a62e365bf957e8ca2d2d920e85618" gracePeriod=30
Nov 02 12:38:20.343695 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:20.343671    4464 generic.go:356] "Generic (PLEG): container finished" podID="b09daaa3-1e6f-4bb0-9437-6d440e085446" containerID="0402b9ca56529f05ebf22e00676cdf6e8a45fa059e8429a4bb6251211c9ea73b" exitCode=0
Nov 02 12:38:20.344039 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:20.343723    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln" event={"ID":"b09daaa3-1e6f-4bb0-9437-6d440e085446","Type":"ContainerDied","Data":"0402b9ca56529f05ebf22e00676cdf6e8a45fa059e8429a4bb6251211c9ea73b"}
Nov 02 12:38:20.344039 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:20.343779    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln" event={"ID":"b09daaa3-1e6f-4bb0-9437-6d440e085446","Type":"ContainerStarted","Data":"384a7094b6e3235f215ab153522a7b987df5e673e99801faccdd71c28ed55a50"}
Nov 02 12:38:21.484573 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:21.484551    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"
Nov 02 12:38:21.489662 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:21.489643    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b09daaa3-1e6f-4bb0-9437-6d440e085446-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b09daaa3-1e6f-4bb0-9437-6d440e085446-maas-api-openshift-ca\") pod \"b09daaa3-1e6f-4bb0-9437-6d440e085446\" (UID: \"b09daaa3-1e6f-4bb0-9437-6d440e085446\") "
Nov 02 12:38:21.489772 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:21.489754    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b09daaa3-1e6f-4bb0-9437-6d440e085446-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b09daaa3-1e6f-4bb0-9437-6d440e085446-maas-api-xks-ca\") pod \"b09daaa3-1e6f-4bb0-9437-6d440e085446\" (UID: \"b09daaa3-1e6f-4bb0-9437-6d440e085446\") "
Nov 02 12:38:21.489872 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:21.489855    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b09daaa3-1e6f-4bb0-9437-6d440e085446-kube-api-access-8twhm\" (UniqueName: \"kubernetes.io/projected/b09daaa3-1e6f-4bb0-9437-6d440e085446-kube-api-access-8twhm\") pod \"b09daaa3-1e6f-4bb0-9437-6d440e085446\" (UID: \"b09daaa3-1e6f-4bb0-9437-6d440e085446\") "
Nov 02 12:38:21.490046 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:21.490023    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b09daaa3-1e6f-4bb0-9437-6d440e085446-maas-api-openshift-ca" pod "b09daaa3-1e6f-4bb0-9437-6d440e085446" (UID: "b09daaa3-1e6f-4bb0-9437-6d440e085446"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:38:21.490207 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:21.490193    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b09daaa3-1e6f-4bb0-9437-6d440e085446-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:21.491681 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:21.491654    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b09daaa3-1e6f-4bb0-9437-6d440e085446-maas-api-xks-ca" pod "b09daaa3-1e6f-4bb0-9437-6d440e085446" (UID: "b09daaa3-1e6f-4bb0-9437-6d440e085446"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:38:21.491785 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:21.491732    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b09daaa3-1e6f-4bb0-9437-6d440e085446-kube-api-access-8twhm" pod "b09daaa3-1e6f-4bb0-9437-6d440e085446" (UID: "b09daaa3-1e6f-4bb0-9437-6d440e085446"). InnerVolumeSpecName "kube-api-access-8twhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:38:21.591486 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:21.591461    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b09daaa3-1e6f-4bb0-9437-6d440e085446-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:21.591486 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:21.591486    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8twhm\" (UniqueName: \"kubernetes.io/projected/b09daaa3-1e6f-4bb0-9437-6d440e085446-kube-api-access-8twhm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:22.357003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:22.356958    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"
Nov 02 12:38:22.357003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:22.356957    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln" event={"ID":"b09daaa3-1e6f-4bb0-9437-6d440e085446","Type":"ContainerDied","Data":"384a7094b6e3235f215ab153522a7b987df5e673e99801faccdd71c28ed55a50"}
Nov 02 12:38:22.357272 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:22.357036    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="384a7094b6e3235f215ab153522a7b987df5e673e99801faccdd71c28ed55a50"
Nov 02 12:38:24.618501 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.618469    4464 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 02 12:38:24.618874 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.618520    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb" podUID="4b684afa-613a-43b5-b10d-0f0ece8da04c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 02 12:38:24.773332 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.773307    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:38:24.819425 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.819345    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-workload-certs\") pod \"4b684afa-613a-43b5-b10d-0f0ece8da04c\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") "
Nov 02 12:38:24.819425 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.819392    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-workload-socket\") pod \"4b684afa-613a-43b5-b10d-0f0ece8da04c\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") "
Nov 02 12:38:24.819649 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.819432    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-token\" (UniqueName: \"kubernetes.io/projected/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-token\") pod \"4b684afa-613a-43b5-b10d-0f0ece8da04c\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") "
Nov 02 12:38:24.819649 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.819476    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-data\") pod \"4b684afa-613a-43b5-b10d-0f0ece8da04c\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") "
Nov 02 12:38:24.819649 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.819538    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-envoy\") pod \"4b684afa-613a-43b5-b10d-0f0ece8da04c\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") "
Nov 02 12:38:24.819649 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.819579    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-podinfo\") pod \"4b684afa-613a-43b5-b10d-0f0ece8da04c\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") "
Nov 02 12:38:24.819649 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.819624    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b684afa-613a-43b5-b10d-0f0ece8da04c-kube-api-access-tblrj\" (UniqueName: \"kubernetes.io/projected/4b684afa-613a-43b5-b10d-0f0ece8da04c-kube-api-access-tblrj\") pod \"4b684afa-613a-43b5-b10d-0f0ece8da04c\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") "
Nov 02 12:38:24.819924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.819619    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-workload-certs" pod "4b684afa-613a-43b5-b10d-0f0ece8da04c" (UID: "4b684afa-613a-43b5-b10d-0f0ece8da04c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:38:24.819924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.819680    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-credential-socket\") pod \"4b684afa-613a-43b5-b10d-0f0ece8da04c\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") "
Nov 02 12:38:24.819924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.819691    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-workload-socket" pod "4b684afa-613a-43b5-b10d-0f0ece8da04c" (UID: "4b684afa-613a-43b5-b10d-0f0ece8da04c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:38:24.819924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.819725    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b684afa-613a-43b5-b10d-0f0ece8da04c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b684afa-613a-43b5-b10d-0f0ece8da04c-istiod-ca-cert\") pod \"4b684afa-613a-43b5-b10d-0f0ece8da04c\" (UID: \"4b684afa-613a-43b5-b10d-0f0ece8da04c\") "
Nov 02 12:38:24.819924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.819790    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-data" pod "4b684afa-613a-43b5-b10d-0f0ece8da04c" (UID: "4b684afa-613a-43b5-b10d-0f0ece8da04c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:38:24.820220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.819968    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-credential-socket" pod "4b684afa-613a-43b5-b10d-0f0ece8da04c" (UID: "4b684afa-613a-43b5-b10d-0f0ece8da04c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:38:24.820220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.820140    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b684afa-613a-43b5-b10d-0f0ece8da04c-istiod-ca-cert" pod "4b684afa-613a-43b5-b10d-0f0ece8da04c" (UID: "4b684afa-613a-43b5-b10d-0f0ece8da04c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:38:24.820220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.820204    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:24.820337 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.820224    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:24.820337 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.820238    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:24.820337 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.820252    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:24.821719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.821695    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b684afa-613a-43b5-b10d-0f0ece8da04c-kube-api-access-tblrj" pod "4b684afa-613a-43b5-b10d-0f0ece8da04c" (UID: "4b684afa-613a-43b5-b10d-0f0ece8da04c"). InnerVolumeSpecName "kube-api-access-tblrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:38:24.821835 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.821745    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-podinfo" pod "4b684afa-613a-43b5-b10d-0f0ece8da04c" (UID: "4b684afa-613a-43b5-b10d-0f0ece8da04c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:38:24.822015 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.821990    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-token" pod "4b684afa-613a-43b5-b10d-0f0ece8da04c" (UID: "4b684afa-613a-43b5-b10d-0f0ece8da04c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:38:24.822098 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.822029    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-envoy" pod "4b684afa-613a-43b5-b10d-0f0ece8da04c" (UID: "4b684afa-613a-43b5-b10d-0f0ece8da04c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:38:24.921278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.921245    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:24.921278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.921274    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:24.921278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.921285    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tblrj\" (UniqueName: \"kubernetes.io/projected/4b684afa-613a-43b5-b10d-0f0ece8da04c-kube-api-access-tblrj\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:24.921510 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.921295    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b684afa-613a-43b5-b10d-0f0ece8da04c-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:24.921510 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:24.921303    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b684afa-613a-43b5-b10d-0f0ece8da04c-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:25.373378 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:25.373342    4464 generic.go:356] "Generic (PLEG): container finished" podID="4b684afa-613a-43b5-b10d-0f0ece8da04c" containerID="da2129fb600ab49a1312806990197fed9e3a62e365bf957e8ca2d2d920e85618" exitCode=0
Nov 02 12:38:25.373553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:25.373395    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"
Nov 02 12:38:25.373553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:25.373431    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb" event={"ID":"4b684afa-613a-43b5-b10d-0f0ece8da04c","Type":"ContainerDied","Data":"da2129fb600ab49a1312806990197fed9e3a62e365bf957e8ca2d2d920e85618"}
Nov 02 12:38:25.373553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:25.373491    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb" event={"ID":"4b684afa-613a-43b5-b10d-0f0ece8da04c","Type":"ContainerDied","Data":"f17432f849c61a5a1276a1f7df1745c3425d325db4f45207319a2d70a334ea5d"}
Nov 02 12:38:25.373553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:25.373522    4464 scope.go:122] "RemoveContainer" containerID="da2129fb600ab49a1312806990197fed9e3a62e365bf957e8ca2d2d920e85618"
Nov 02 12:38:25.383936 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:25.383919    4464 scope.go:122] "RemoveContainer" containerID="da2129fb600ab49a1312806990197fed9e3a62e365bf957e8ca2d2d920e85618"
Nov 02 12:38:25.384249 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:38:25.384230    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da2129fb600ab49a1312806990197fed9e3a62e365bf957e8ca2d2d920e85618\": container with ID starting with da2129fb600ab49a1312806990197fed9e3a62e365bf957e8ca2d2d920e85618 not found: ID does not exist" containerID="da2129fb600ab49a1312806990197fed9e3a62e365bf957e8ca2d2d920e85618"
Nov 02 12:38:25.384385 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:25.384259    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da2129fb600ab49a1312806990197fed9e3a62e365bf957e8ca2d2d920e85618"} err="failed to get container status \"da2129fb600ab49a1312806990197fed9e3a62e365bf957e8ca2d2d920e85618\": rpc error: code = NotFound desc = could not find container \"da2129fb600ab49a1312806990197fed9e3a62e365bf957e8ca2d2d920e85618\": container with ID starting with da2129fb600ab49a1312806990197fed9e3a62e365bf957e8ca2d2d920e85618 not found: ID does not exist"
Nov 02 12:38:25.547347 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:25.547308    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"]
Nov 02 12:38:25.553355 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:25.553328    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-t6nlb"]
Nov 02 12:38:25.581583 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:25.581555    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b684afa-613a-43b5-b10d-0f0ece8da04c" path="/var/lib/kubelet/pods/4b684afa-613a-43b5-b10d-0f0ece8da04c/volumes"
Nov 02 12:38:26.324428 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:26.324385    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67" podUID="27d181d3-3802-44f7-ace0-e66f40244776" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:38:27.491388 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:27.491356    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"]
Nov 02 12:38:28.292308 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:38:28.292275    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:38:29.223321 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:29.223284    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r" podUID="fc81545d-7d40-4b73-9e60-69d2348cfddd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:38:30.574056 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:30.574026    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:38:31.492814 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.492780    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"]
Nov 02 12:38:31.493303 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.493288    4464 cpu_manager.go:414] "containerMap: removing container" podUID="4b684afa-613a-43b5-b10d-0f0ece8da04c" containerName="istio-proxy"
Nov 02 12:38:31.493352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.493304    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b684afa-613a-43b5-b10d-0f0ece8da04c" containerName="istio-proxy"
Nov 02 12:38:31.493352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.493316    4464 cpu_manager.go:414] "containerMap: removing container" podUID="b09daaa3-1e6f-4bb0-9437-6d440e085446" containerName="revoke-keys"
Nov 02 12:38:31.493352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.493321    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b09daaa3-1e6f-4bb0-9437-6d440e085446" containerName="revoke-keys"
Nov 02 12:38:31.493465 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.493397    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b684afa-613a-43b5-b10d-0f0ece8da04c" containerName="istio-proxy"
Nov 02 12:38:31.493465 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.493405    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="b09daaa3-1e6f-4bb0-9437-6d440e085446" containerName="revoke-keys"
Nov 02 12:38:31.497567 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.497550    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"
Nov 02 12:38:31.591701 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.591665    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1514196f-2354-4e14-a8b6-939cd6f1346b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cde896e-jgd98\" (UID: \"1514196f-2354-4e14-a8b6-939cd6f1346b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"
Nov 02 12:38:31.591701 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.591701    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1514196f-2354-4e14-a8b6-939cd6f1346b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cde896e-jgd98\" (UID: \"1514196f-2354-4e14-a8b6-939cd6f1346b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"
Nov 02 12:38:31.592128 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.591724    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dcd5\" (UniqueName: \"kubernetes.io/projected/1514196f-2354-4e14-a8b6-939cd6f1346b-kube-api-access-5dcd5\") pod \"maas-api-revoke-sub-3cde896e-jgd98\" (UID: \"1514196f-2354-4e14-a8b6-939cd6f1346b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"
Nov 02 12:38:31.602822 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.602789    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:38:31.602822 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.602815    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:38:31.603033 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.602827    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:38:31.603033 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.602904    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:38:31.605098 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.605080    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"]
Nov 02 12:38:31.692330 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.692307    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1514196f-2354-4e14-a8b6-939cd6f1346b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cde896e-jgd98\" (UID: \"1514196f-2354-4e14-a8b6-939cd6f1346b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"
Nov 02 12:38:31.692476 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.692336    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1514196f-2354-4e14-a8b6-939cd6f1346b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cde896e-jgd98\" (UID: \"1514196f-2354-4e14-a8b6-939cd6f1346b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"
Nov 02 12:38:31.692476 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.692360    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dcd5\" (UniqueName: \"kubernetes.io/projected/1514196f-2354-4e14-a8b6-939cd6f1346b-kube-api-access-5dcd5\") pod \"maas-api-revoke-sub-3cde896e-jgd98\" (UID: \"1514196f-2354-4e14-a8b6-939cd6f1346b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"
Nov 02 12:38:31.692922 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.692902    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1514196f-2354-4e14-a8b6-939cd6f1346b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cde896e-jgd98\" (UID: \"1514196f-2354-4e14-a8b6-939cd6f1346b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"
Nov 02 12:38:31.694658 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.694640    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1514196f-2354-4e14-a8b6-939cd6f1346b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cde896e-jgd98\" (UID: \"1514196f-2354-4e14-a8b6-939cd6f1346b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"
Nov 02 12:38:31.754564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.754505    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dcd5\" (UniqueName: \"kubernetes.io/projected/1514196f-2354-4e14-a8b6-939cd6f1346b-kube-api-access-5dcd5\") pod \"maas-api-revoke-sub-3cde896e-jgd98\" (UID: \"1514196f-2354-4e14-a8b6-939cd6f1346b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"
Nov 02 12:38:31.811213 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:31.811188    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"
Nov 02 12:38:32.204146 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:32.204124    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"]
Nov 02 12:38:32.206963 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:38:32.206922    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1514196f_2354_4e14_a8b6_939cd6f1346b.slice/crio-547aa23e09d6a18283ce89d57b59e39aef174725cf06068b4f9bbed27c33d684 WatchSource:0}: Error finding container 547aa23e09d6a18283ce89d57b59e39aef174725cf06068b4f9bbed27c33d684: Status 404 returned error can't find the container with id 547aa23e09d6a18283ce89d57b59e39aef174725cf06068b4f9bbed27c33d684
Nov 02 12:38:32.411165 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:32.411132    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98" event={"ID":"1514196f-2354-4e14-a8b6-939cd6f1346b","Type":"ContainerStarted","Data":"547aa23e09d6a18283ce89d57b59e39aef174725cf06068b4f9bbed27c33d684"}
Nov 02 12:38:32.491792 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:32.491749    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq" podUID="ab65a1ba-c07c-4930-ac5b-e80eaadfe184" containerName="payload-processing" containerID="cri-o://0f84511edd9f7198fa05d8bc997d46b8b72d88df495481fc2a25bf0ddf4687a1" gracePeriod=25
Nov 02 12:38:33.417965 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:33.417928    4464 generic.go:356] "Generic (PLEG): container finished" podID="1514196f-2354-4e14-a8b6-939cd6f1346b" containerID="02dd95c792a336cd1b0e369264ec628847e654e90f592ffd8c42471f12efb470" exitCode=0
Nov 02 12:38:33.418372 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:33.418018    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98" event={"ID":"1514196f-2354-4e14-a8b6-939cd6f1346b","Type":"ContainerDied","Data":"02dd95c792a336cd1b0e369264ec628847e654e90f592ffd8c42471f12efb470"}
Nov 02 12:38:34.554233 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:34.554205    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"
Nov 02 12:38:34.620544 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:34.620509    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1514196f-2354-4e14-a8b6-939cd6f1346b-kube-api-access-5dcd5\" (UniqueName: \"kubernetes.io/projected/1514196f-2354-4e14-a8b6-939cd6f1346b-kube-api-access-5dcd5\") pod \"1514196f-2354-4e14-a8b6-939cd6f1346b\" (UID: \"1514196f-2354-4e14-a8b6-939cd6f1346b\") "
Nov 02 12:38:34.620732 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:34.620583    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1514196f-2354-4e14-a8b6-939cd6f1346b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1514196f-2354-4e14-a8b6-939cd6f1346b-maas-api-openshift-ca\") pod \"1514196f-2354-4e14-a8b6-939cd6f1346b\" (UID: \"1514196f-2354-4e14-a8b6-939cd6f1346b\") "
Nov 02 12:38:34.620732 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:34.620630    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1514196f-2354-4e14-a8b6-939cd6f1346b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1514196f-2354-4e14-a8b6-939cd6f1346b-maas-api-xks-ca\") pod \"1514196f-2354-4e14-a8b6-939cd6f1346b\" (UID: \"1514196f-2354-4e14-a8b6-939cd6f1346b\") "
Nov 02 12:38:34.621049 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:34.621022    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1514196f-2354-4e14-a8b6-939cd6f1346b-maas-api-openshift-ca" pod "1514196f-2354-4e14-a8b6-939cd6f1346b" (UID: "1514196f-2354-4e14-a8b6-939cd6f1346b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:38:34.622637 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:34.622608    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1514196f-2354-4e14-a8b6-939cd6f1346b-maas-api-xks-ca" pod "1514196f-2354-4e14-a8b6-939cd6f1346b" (UID: "1514196f-2354-4e14-a8b6-939cd6f1346b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:38:34.622726 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:34.622687    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1514196f-2354-4e14-a8b6-939cd6f1346b-kube-api-access-5dcd5" pod "1514196f-2354-4e14-a8b6-939cd6f1346b" (UID: "1514196f-2354-4e14-a8b6-939cd6f1346b"). InnerVolumeSpecName "kube-api-access-5dcd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:38:34.722320 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:34.722291    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dcd5\" (UniqueName: \"kubernetes.io/projected/1514196f-2354-4e14-a8b6-939cd6f1346b-kube-api-access-5dcd5\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:34.722320 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:34.722321    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1514196f-2354-4e14-a8b6-939cd6f1346b-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:34.722501 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:34.722331    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1514196f-2354-4e14-a8b6-939cd6f1346b-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:34.782383 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:34.782347    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"]
Nov 02 12:38:34.782621 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:34.782593    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d" podUID="9d39d963-d570-44cc-96fd-a4efee1e45fc" containerName="istio-proxy" containerID="cri-o://a41d4fb954400960308b10726dca8e523cc238e6c5170d13757177deda8f33dd" gracePeriod=30
Nov 02 12:38:35.428596 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:35.428570    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"
Nov 02 12:38:35.430008 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:35.429967    4464 generic.go:356] "Generic (PLEG): container finished" podID="fc81545d-7d40-4b73-9e60-69d2348cfddd" containerID="afb4c368ae4bf0971015cca0c47994b11707388ec7b58fa35d6c8b5f746c2bbd" exitCode=0
Nov 02 12:38:35.430151 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:35.430021    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r" event={"ID":"fc81545d-7d40-4b73-9e60-69d2348cfddd","Type":"ContainerDied","Data":"afb4c368ae4bf0971015cca0c47994b11707388ec7b58fa35d6c8b5f746c2bbd"}
Nov 02 12:38:35.430151 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:35.430088    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r" event={"ID":"fc81545d-7d40-4b73-9e60-69d2348cfddd","Type":"ContainerDied","Data":"1805afc61f25c045449553247c7f46a1c20f8ebe34e01293fe78d4597a033540"}
Nov 02 12:38:35.430151 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:35.430126    4464 scope.go:122] "RemoveContainer" containerID="afb4c368ae4bf0971015cca0c47994b11707388ec7b58fa35d6c8b5f746c2bbd"
Nov 02 12:38:35.431468 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:35.431324    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"
Nov 02 12:38:35.431468 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:35.431327    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98" event={"ID":"1514196f-2354-4e14-a8b6-939cd6f1346b","Type":"ContainerDied","Data":"547aa23e09d6a18283ce89d57b59e39aef174725cf06068b4f9bbed27c33d684"}
Nov 02 12:38:35.431468 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:35.431367    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="547aa23e09d6a18283ce89d57b59e39aef174725cf06068b4f9bbed27c33d684"
Nov 02 12:38:35.439709 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:35.439658    4464 scope.go:122] "RemoveContainer" containerID="afb4c368ae4bf0971015cca0c47994b11707388ec7b58fa35d6c8b5f746c2bbd"
Nov 02 12:38:35.439956 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:38:35.439936    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afb4c368ae4bf0971015cca0c47994b11707388ec7b58fa35d6c8b5f746c2bbd\": container with ID starting with afb4c368ae4bf0971015cca0c47994b11707388ec7b58fa35d6c8b5f746c2bbd not found: ID does not exist" containerID="afb4c368ae4bf0971015cca0c47994b11707388ec7b58fa35d6c8b5f746c2bbd"
Nov 02 12:38:35.440075 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:35.439963    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afb4c368ae4bf0971015cca0c47994b11707388ec7b58fa35d6c8b5f746c2bbd"} err="failed to get container status \"afb4c368ae4bf0971015cca0c47994b11707388ec7b58fa35d6c8b5f746c2bbd\": rpc error: code = NotFound desc = could not find container \"afb4c368ae4bf0971015cca0c47994b11707388ec7b58fa35d6c8b5f746c2bbd\": container with ID starting with afb4c368ae4bf0971015cca0c47994b11707388ec7b58fa35d6c8b5f746c2bbd not found: ID does not exist"
Nov 02 12:38:35.528798 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:35.528770    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc81545d-7d40-4b73-9e60-69d2348cfddd-kube-api-access-z9xk4\" (UniqueName: \"kubernetes.io/projected/fc81545d-7d40-4b73-9e60-69d2348cfddd-kube-api-access-z9xk4\") pod \"fc81545d-7d40-4b73-9e60-69d2348cfddd\" (UID: \"fc81545d-7d40-4b73-9e60-69d2348cfddd\") "
Nov 02 12:38:35.528903 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:35.528809    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc81545d-7d40-4b73-9e60-69d2348cfddd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc81545d-7d40-4b73-9e60-69d2348cfddd-plugins-config-volume\") pod \"fc81545d-7d40-4b73-9e60-69d2348cfddd\" (UID: \"fc81545d-7d40-4b73-9e60-69d2348cfddd\") "
Nov 02 12:38:35.529218 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:35.529198    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc81545d-7d40-4b73-9e60-69d2348cfddd-plugins-config-volume" pod "fc81545d-7d40-4b73-9e60-69d2348cfddd" (UID: "fc81545d-7d40-4b73-9e60-69d2348cfddd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:38:35.530751 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:35.530732    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc81545d-7d40-4b73-9e60-69d2348cfddd-kube-api-access-z9xk4" pod "fc81545d-7d40-4b73-9e60-69d2348cfddd" (UID: "fc81545d-7d40-4b73-9e60-69d2348cfddd"). InnerVolumeSpecName "kube-api-access-z9xk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:38:35.630380 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:35.630351    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9xk4\" (UniqueName: \"kubernetes.io/projected/fc81545d-7d40-4b73-9e60-69d2348cfddd-kube-api-access-z9xk4\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:35.630380 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:35.630377    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc81545d-7d40-4b73-9e60-69d2348cfddd-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:36.322539 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:36.322496    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67" podUID="27d181d3-3802-44f7-ace0-e66f40244776" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:38:36.445030 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:36.444995    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"
Nov 02 12:38:36.612169 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:36.612088    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"]
Nov 02 12:38:36.618180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:36.618148    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-dd087a-769f959885-x622r"]
Nov 02 12:38:37.580393 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:37.580361    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc81545d-7d40-4b73-9e60-69d2348cfddd" path="/var/lib/kubelet/pods/fc81545d-7d40-4b73-9e60-69d2348cfddd/volumes"
Nov 02 12:38:37.706763 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:37.706731    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq" podUID="ab65a1ba-c07c-4930-ac5b-e80eaadfe184" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:38:40.043953 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.043929    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:38:40.174675 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.174600    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-podinfo\") pod \"9d39d963-d570-44cc-96fd-a4efee1e45fc\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") "
Nov 02 12:38:40.174675 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.174655    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-envoy\") pod \"9d39d963-d570-44cc-96fd-a4efee1e45fc\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") "
Nov 02 12:38:40.174902 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.174677    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d39d963-d570-44cc-96fd-a4efee1e45fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d39d963-d570-44cc-96fd-a4efee1e45fc-istiod-ca-cert\") pod \"9d39d963-d570-44cc-96fd-a4efee1e45fc\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") "
Nov 02 12:38:40.174902 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.174709    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-credential-socket\") pod \"9d39d963-d570-44cc-96fd-a4efee1e45fc\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") "
Nov 02 12:38:40.174902 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.174778    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-token\" (UniqueName: \"kubernetes.io/projected/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-token\") pod \"9d39d963-d570-44cc-96fd-a4efee1e45fc\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") "
Nov 02 12:38:40.174902 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.174814    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-data\") pod \"9d39d963-d570-44cc-96fd-a4efee1e45fc\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") "
Nov 02 12:38:40.174902 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.174841    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-workload-certs\") pod \"9d39d963-d570-44cc-96fd-a4efee1e45fc\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") "
Nov 02 12:38:40.174902 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.174877    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d39d963-d570-44cc-96fd-a4efee1e45fc-kube-api-access-nkz78\" (UniqueName: \"kubernetes.io/projected/9d39d963-d570-44cc-96fd-a4efee1e45fc-kube-api-access-nkz78\") pod \"9d39d963-d570-44cc-96fd-a4efee1e45fc\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") "
Nov 02 12:38:40.175269 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.174948    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-workload-socket\") pod \"9d39d963-d570-44cc-96fd-a4efee1e45fc\" (UID: \"9d39d963-d570-44cc-96fd-a4efee1e45fc\") "
Nov 02 12:38:40.175269 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.175088    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-credential-socket" pod "9d39d963-d570-44cc-96fd-a4efee1e45fc" (UID: "9d39d963-d570-44cc-96fd-a4efee1e45fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:38:40.175269 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.175206    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-workload-certs" pod "9d39d963-d570-44cc-96fd-a4efee1e45fc" (UID: "9d39d963-d570-44cc-96fd-a4efee1e45fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:38:40.175269 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.175238    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d39d963-d570-44cc-96fd-a4efee1e45fc-istiod-ca-cert" pod "9d39d963-d570-44cc-96fd-a4efee1e45fc" (UID: "9d39d963-d570-44cc-96fd-a4efee1e45fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:38:40.175495 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.175269    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-workload-socket" pod "9d39d963-d570-44cc-96fd-a4efee1e45fc" (UID: "9d39d963-d570-44cc-96fd-a4efee1e45fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:38:40.175495 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.175401    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:40.175495 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.175418    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d39d963-d570-44cc-96fd-a4efee1e45fc-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:40.175495 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.175432    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:40.175495 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.175435    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-data" pod "9d39d963-d570-44cc-96fd-a4efee1e45fc" (UID: "9d39d963-d570-44cc-96fd-a4efee1e45fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:38:40.175495 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.175444    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:40.176997 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.176941    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-envoy" pod "9d39d963-d570-44cc-96fd-a4efee1e45fc" (UID: "9d39d963-d570-44cc-96fd-a4efee1e45fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:38:40.177125 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.177016    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-token" pod "9d39d963-d570-44cc-96fd-a4efee1e45fc" (UID: "9d39d963-d570-44cc-96fd-a4efee1e45fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:38:40.177125 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.177028    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-podinfo" pod "9d39d963-d570-44cc-96fd-a4efee1e45fc" (UID: "9d39d963-d570-44cc-96fd-a4efee1e45fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:38:40.177125 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.177045    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d39d963-d570-44cc-96fd-a4efee1e45fc-kube-api-access-nkz78" pod "9d39d963-d570-44cc-96fd-a4efee1e45fc" (UID: "9d39d963-d570-44cc-96fd-a4efee1e45fc"). InnerVolumeSpecName "kube-api-access-nkz78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:38:40.276889 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.276856    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:40.276889 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.276882    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:40.276889 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.276891    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:40.277130 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.276899    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d39d963-d570-44cc-96fd-a4efee1e45fc-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:40.277130 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.276908    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkz78\" (UniqueName: \"kubernetes.io/projected/9d39d963-d570-44cc-96fd-a4efee1e45fc-kube-api-access-nkz78\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:40.468403 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.468374    4464 generic.go:356] "Generic (PLEG): container finished" podID="9d39d963-d570-44cc-96fd-a4efee1e45fc" containerID="a41d4fb954400960308b10726dca8e523cc238e6c5170d13757177deda8f33dd" exitCode=0
Nov 02 12:38:40.468561 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.468423    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"
Nov 02 12:38:40.468561 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.468430    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d" event={"ID":"9d39d963-d570-44cc-96fd-a4efee1e45fc","Type":"ContainerDied","Data":"a41d4fb954400960308b10726dca8e523cc238e6c5170d13757177deda8f33dd"}
Nov 02 12:38:40.468561 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.468475    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d" event={"ID":"9d39d963-d570-44cc-96fd-a4efee1e45fc","Type":"ContainerDied","Data":"ef7e5983ee81c3e586cec5d61841b85023c7150b6e57221c51a9713af3917a55"}
Nov 02 12:38:40.468561 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.468499    4464 scope.go:122] "RemoveContainer" containerID="a41d4fb954400960308b10726dca8e523cc238e6c5170d13757177deda8f33dd"
Nov 02 12:38:40.478123 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.478103    4464 scope.go:122] "RemoveContainer" containerID="b5870258acae6a70a0cdf5fc7482cf5c13d061212278229a91117c59090dbf4f"
Nov 02 12:38:40.486715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.486689    4464 scope.go:122] "RemoveContainer" containerID="a41d4fb954400960308b10726dca8e523cc238e6c5170d13757177deda8f33dd"
Nov 02 12:38:40.487020 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:38:40.486994    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41d4fb954400960308b10726dca8e523cc238e6c5170d13757177deda8f33dd\": container with ID starting with a41d4fb954400960308b10726dca8e523cc238e6c5170d13757177deda8f33dd not found: ID does not exist" containerID="a41d4fb954400960308b10726dca8e523cc238e6c5170d13757177deda8f33dd"
Nov 02 12:38:40.487120 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.487025    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41d4fb954400960308b10726dca8e523cc238e6c5170d13757177deda8f33dd"} err="failed to get container status \"a41d4fb954400960308b10726dca8e523cc238e6c5170d13757177deda8f33dd\": rpc error: code = NotFound desc = could not find container \"a41d4fb954400960308b10726dca8e523cc238e6c5170d13757177deda8f33dd\": container with ID starting with a41d4fb954400960308b10726dca8e523cc238e6c5170d13757177deda8f33dd not found: ID does not exist"
Nov 02 12:38:40.487120 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.487042    4464 scope.go:122] "RemoveContainer" containerID="b5870258acae6a70a0cdf5fc7482cf5c13d061212278229a91117c59090dbf4f"
Nov 02 12:38:40.487290 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:38:40.487274    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5870258acae6a70a0cdf5fc7482cf5c13d061212278229a91117c59090dbf4f\": container with ID starting with b5870258acae6a70a0cdf5fc7482cf5c13d061212278229a91117c59090dbf4f not found: ID does not exist" containerID="b5870258acae6a70a0cdf5fc7482cf5c13d061212278229a91117c59090dbf4f"
Nov 02 12:38:40.487336 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.487295    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5870258acae6a70a0cdf5fc7482cf5c13d061212278229a91117c59090dbf4f"} err="failed to get container status \"b5870258acae6a70a0cdf5fc7482cf5c13d061212278229a91117c59090dbf4f\": rpc error: code = NotFound desc = could not find container \"b5870258acae6a70a0cdf5fc7482cf5c13d061212278229a91117c59090dbf4f\": container with ID starting with b5870258acae6a70a0cdf5fc7482cf5c13d061212278229a91117c59090dbf4f not found: ID does not exist"
Nov 02 12:38:40.638021 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.637994    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"]
Nov 02 12:38:40.645827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:40.645801    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-tdk2d"]
Nov 02 12:38:41.580235 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:41.580204    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d39d963-d570-44cc-96fd-a4efee1e45fc" path="/var/lib/kubelet/pods/9d39d963-d570-44cc-96fd-a4efee1e45fc/volumes"
Nov 02 12:38:43.213892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.213853    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"]
Nov 02 12:38:43.214368 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.214354    4464 cpu_manager.go:414] "containerMap: removing container" podUID="1514196f-2354-4e14-a8b6-939cd6f1346b" containerName="revoke-keys"
Nov 02 12:38:43.214418 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.214369    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1514196f-2354-4e14-a8b6-939cd6f1346b" containerName="revoke-keys"
Nov 02 12:38:43.214418 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.214387    4464 cpu_manager.go:414] "containerMap: removing container" podUID="9d39d963-d570-44cc-96fd-a4efee1e45fc" containerName="istio-proxy"
Nov 02 12:38:43.214418 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.214392    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d39d963-d570-44cc-96fd-a4efee1e45fc" containerName="istio-proxy"
Nov 02 12:38:43.214418 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.214408    4464 cpu_manager.go:414] "containerMap: removing container" podUID="fc81545d-7d40-4b73-9e60-69d2348cfddd" containerName="payload-processing"
Nov 02 12:38:43.214418 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.214418    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc81545d-7d40-4b73-9e60-69d2348cfddd" containerName="payload-processing"
Nov 02 12:38:43.214570 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.214428    4464 cpu_manager.go:414] "containerMap: removing container" podUID="9d39d963-d570-44cc-96fd-a4efee1e45fc" containerName="istio-proxy"
Nov 02 12:38:43.214570 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.214433    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d39d963-d570-44cc-96fd-a4efee1e45fc" containerName="istio-proxy"
Nov 02 12:38:43.214570 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.214513    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc81545d-7d40-4b73-9e60-69d2348cfddd" containerName="payload-processing"
Nov 02 12:38:43.214570 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.214525    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d39d963-d570-44cc-96fd-a4efee1e45fc" containerName="istio-proxy"
Nov 02 12:38:43.214570 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.214535    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="1514196f-2354-4e14-a8b6-939cd6f1346b" containerName="revoke-keys"
Nov 02 12:38:43.217654 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.217638    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"
Nov 02 12:38:43.272518 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.272490    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:38:43.302191 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.302169    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07429504-309b-40e8-b1d5-9c7bde3b683d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42b888e2-whh7m\" (UID: \"07429504-309b-40e8-b1d5-9c7bde3b683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"
Nov 02 12:38:43.302349 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.302207    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07429504-309b-40e8-b1d5-9c7bde3b683d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42b888e2-whh7m\" (UID: \"07429504-309b-40e8-b1d5-9c7bde3b683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"
Nov 02 12:38:43.302425 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.302343    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwvdb\" (UniqueName: \"kubernetes.io/projected/07429504-309b-40e8-b1d5-9c7bde3b683d-kube-api-access-wwvdb\") pod \"maas-api-revoke-sub-42b888e2-whh7m\" (UID: \"07429504-309b-40e8-b1d5-9c7bde3b683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"
Nov 02 12:38:43.323277 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.323256    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:38:43.323416 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.323310    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:38:43.323482 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.323445    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:38:43.325634 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.325609    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"]
Nov 02 12:38:43.403834 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.403810    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwvdb\" (UniqueName: \"kubernetes.io/projected/07429504-309b-40e8-b1d5-9c7bde3b683d-kube-api-access-wwvdb\") pod \"maas-api-revoke-sub-42b888e2-whh7m\" (UID: \"07429504-309b-40e8-b1d5-9c7bde3b683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"
Nov 02 12:38:43.403968 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.403886    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07429504-309b-40e8-b1d5-9c7bde3b683d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42b888e2-whh7m\" (UID: \"07429504-309b-40e8-b1d5-9c7bde3b683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"
Nov 02 12:38:43.403968 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.403922    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07429504-309b-40e8-b1d5-9c7bde3b683d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42b888e2-whh7m\" (UID: \"07429504-309b-40e8-b1d5-9c7bde3b683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"
Nov 02 12:38:43.404519 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.404501    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07429504-309b-40e8-b1d5-9c7bde3b683d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42b888e2-whh7m\" (UID: \"07429504-309b-40e8-b1d5-9c7bde3b683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"
Nov 02 12:38:43.406281 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.406261    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07429504-309b-40e8-b1d5-9c7bde3b683d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42b888e2-whh7m\" (UID: \"07429504-309b-40e8-b1d5-9c7bde3b683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"
Nov 02 12:38:43.465505 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.465448    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwvdb\" (UniqueName: \"kubernetes.io/projected/07429504-309b-40e8-b1d5-9c7bde3b683d-kube-api-access-wwvdb\") pod \"maas-api-revoke-sub-42b888e2-whh7m\" (UID: \"07429504-309b-40e8-b1d5-9c7bde3b683d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"
Nov 02 12:38:43.530184 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.530160    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"
Nov 02 12:38:43.900855 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:43.900829    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"]
Nov 02 12:38:43.902660 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:38:43.902615    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07429504_309b_40e8_b1d5_9c7bde3b683d.slice/crio-f6df33e86e96bcc9eae8ab4c4139b6c2cdad146e540944139785edce8dfd62f3 WatchSource:0}: Error finding container f6df33e86e96bcc9eae8ab4c4139b6c2cdad146e540944139785edce8dfd62f3: Status 404 returned error can't find the container with id f6df33e86e96bcc9eae8ab4c4139b6c2cdad146e540944139785edce8dfd62f3
Nov 02 12:38:44.495043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:44.495010    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m" event={"ID":"07429504-309b-40e8-b1d5-9c7bde3b683d","Type":"ContainerStarted","Data":"19417c8ceb12073e2d0e0ea50c12d4121325e154d5cd3ae774a69b529e2b83d2"}
Nov 02 12:38:44.495514 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:44.495072    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m" event={"ID":"07429504-309b-40e8-b1d5-9c7bde3b683d","Type":"ContainerStarted","Data":"f6df33e86e96bcc9eae8ab4c4139b6c2cdad146e540944139785edce8dfd62f3"}
Nov 02 12:38:44.613069 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:44.613019    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m" podStartSLOduration=1.613006551 podStartE2EDuration="1.613006551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:44.612875086 +0000 UTC m=+2553.176583495" watchObservedRunningTime="2026-11-02 12:38:44.613006551 +0000 UTC m=+2553.176714964"
Nov 02 12:38:45.501884 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:45.501854    4464 generic.go:356] "Generic (PLEG): container finished" podID="07429504-309b-40e8-b1d5-9c7bde3b683d" containerID="19417c8ceb12073e2d0e0ea50c12d4121325e154d5cd3ae774a69b529e2b83d2" exitCode=0
Nov 02 12:38:45.502291 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:45.501937    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m" event={"ID":"07429504-309b-40e8-b1d5-9c7bde3b683d","Type":"ContainerDied","Data":"19417c8ceb12073e2d0e0ea50c12d4121325e154d5cd3ae774a69b529e2b83d2"}
Nov 02 12:38:46.322692 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:46.322652    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67" podUID="27d181d3-3802-44f7-ace0-e66f40244776" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:38:46.664079 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:46.664048    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"
Nov 02 12:38:46.731668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:46.731642    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07429504-309b-40e8-b1d5-9c7bde3b683d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07429504-309b-40e8-b1d5-9c7bde3b683d-maas-api-openshift-ca\") pod \"07429504-309b-40e8-b1d5-9c7bde3b683d\" (UID: \"07429504-309b-40e8-b1d5-9c7bde3b683d\") "
Nov 02 12:38:46.731849 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:46.731834    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07429504-309b-40e8-b1d5-9c7bde3b683d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07429504-309b-40e8-b1d5-9c7bde3b683d-maas-api-xks-ca\") pod \"07429504-309b-40e8-b1d5-9c7bde3b683d\" (UID: \"07429504-309b-40e8-b1d5-9c7bde3b683d\") "
Nov 02 12:38:46.731933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:46.731892    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07429504-309b-40e8-b1d5-9c7bde3b683d-kube-api-access-wwvdb\" (UniqueName: \"kubernetes.io/projected/07429504-309b-40e8-b1d5-9c7bde3b683d-kube-api-access-wwvdb\") pod \"07429504-309b-40e8-b1d5-9c7bde3b683d\" (UID: \"07429504-309b-40e8-b1d5-9c7bde3b683d\") "
Nov 02 12:38:46.732073 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:46.732049    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07429504-309b-40e8-b1d5-9c7bde3b683d-maas-api-openshift-ca" pod "07429504-309b-40e8-b1d5-9c7bde3b683d" (UID: "07429504-309b-40e8-b1d5-9c7bde3b683d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:38:46.732216 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:46.732201    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07429504-309b-40e8-b1d5-9c7bde3b683d-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:46.733942 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:46.733921    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07429504-309b-40e8-b1d5-9c7bde3b683d-kube-api-access-wwvdb" pod "07429504-309b-40e8-b1d5-9c7bde3b683d" (UID: "07429504-309b-40e8-b1d5-9c7bde3b683d"). InnerVolumeSpecName "kube-api-access-wwvdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:38:46.734033 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:46.733946    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07429504-309b-40e8-b1d5-9c7bde3b683d-maas-api-xks-ca" pod "07429504-309b-40e8-b1d5-9c7bde3b683d" (UID: "07429504-309b-40e8-b1d5-9c7bde3b683d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:38:46.833072 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:46.833040    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07429504-309b-40e8-b1d5-9c7bde3b683d-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:46.833072 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:46.833067    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwvdb\" (UniqueName: \"kubernetes.io/projected/07429504-309b-40e8-b1d5-9c7bde3b683d-kube-api-access-wwvdb\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:47.512712 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:47.512683    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"
Nov 02 12:38:47.512712 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:47.512690    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m" event={"ID":"07429504-309b-40e8-b1d5-9c7bde3b683d","Type":"ContainerDied","Data":"f6df33e86e96bcc9eae8ab4c4139b6c2cdad146e540944139785edce8dfd62f3"}
Nov 02 12:38:47.512917 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:47.512760    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6df33e86e96bcc9eae8ab4c4139b6c2cdad146e540944139785edce8dfd62f3"
Nov 02 12:38:47.706295 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:47.706258    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq" podUID="ab65a1ba-c07c-4930-ac5b-e80eaadfe184" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:38:52.747575 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:52.747513    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"
Nov 02 12:38:52.887106 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:52.887080    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab65a1ba-c07c-4930-ac5b-e80eaadfe184-kube-api-access-vjv58\" (UniqueName: \"kubernetes.io/projected/ab65a1ba-c07c-4930-ac5b-e80eaadfe184-kube-api-access-vjv58\") pod \"ab65a1ba-c07c-4930-ac5b-e80eaadfe184\" (UID: \"ab65a1ba-c07c-4930-ac5b-e80eaadfe184\") "
Nov 02 12:38:52.887258 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:52.887208    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab65a1ba-c07c-4930-ac5b-e80eaadfe184-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab65a1ba-c07c-4930-ac5b-e80eaadfe184-plugins-config-volume\") pod \"ab65a1ba-c07c-4930-ac5b-e80eaadfe184\" (UID: \"ab65a1ba-c07c-4930-ac5b-e80eaadfe184\") "
Nov 02 12:38:52.887654 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:52.887623    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab65a1ba-c07c-4930-ac5b-e80eaadfe184-plugins-config-volume" pod "ab65a1ba-c07c-4930-ac5b-e80eaadfe184" (UID: "ab65a1ba-c07c-4930-ac5b-e80eaadfe184"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:38:52.889101 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:52.889078    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab65a1ba-c07c-4930-ac5b-e80eaadfe184-kube-api-access-vjv58" pod "ab65a1ba-c07c-4930-ac5b-e80eaadfe184" (UID: "ab65a1ba-c07c-4930-ac5b-e80eaadfe184"). InnerVolumeSpecName "kube-api-access-vjv58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:38:52.988133 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:52.988106    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab65a1ba-c07c-4930-ac5b-e80eaadfe184-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:52.988133 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:52.988130    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjv58\" (UniqueName: \"kubernetes.io/projected/ab65a1ba-c07c-4930-ac5b-e80eaadfe184-kube-api-access-vjv58\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:38:53.553716 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:53.553681    4464 generic.go:356] "Generic (PLEG): container finished" podID="ab65a1ba-c07c-4930-ac5b-e80eaadfe184" containerID="0f84511edd9f7198fa05d8bc997d46b8b72d88df495481fc2a25bf0ddf4687a1" exitCode=0
Nov 02 12:38:53.553870 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:53.553735    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"
Nov 02 12:38:53.553870 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:53.553762    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq" event={"ID":"ab65a1ba-c07c-4930-ac5b-e80eaadfe184","Type":"ContainerDied","Data":"0f84511edd9f7198fa05d8bc997d46b8b72d88df495481fc2a25bf0ddf4687a1"}
Nov 02 12:38:53.553870 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:53.553832    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq" event={"ID":"ab65a1ba-c07c-4930-ac5b-e80eaadfe184","Type":"ContainerDied","Data":"d12153f99a6981edc72e2465218c6e1fbf6c894e3bbd2e131224a2d00c064595"}
Nov 02 12:38:53.553870 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:53.553854    4464 scope.go:122] "RemoveContainer" containerID="0f84511edd9f7198fa05d8bc997d46b8b72d88df495481fc2a25bf0ddf4687a1"
Nov 02 12:38:53.565172 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:53.565152    4464 scope.go:122] "RemoveContainer" containerID="0f84511edd9f7198fa05d8bc997d46b8b72d88df495481fc2a25bf0ddf4687a1"
Nov 02 12:38:53.565492 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:38:53.565467    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f84511edd9f7198fa05d8bc997d46b8b72d88df495481fc2a25bf0ddf4687a1\": container with ID starting with 0f84511edd9f7198fa05d8bc997d46b8b72d88df495481fc2a25bf0ddf4687a1 not found: ID does not exist" containerID="0f84511edd9f7198fa05d8bc997d46b8b72d88df495481fc2a25bf0ddf4687a1"
Nov 02 12:38:53.565554 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:53.565504    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f84511edd9f7198fa05d8bc997d46b8b72d88df495481fc2a25bf0ddf4687a1"} err="failed to get container status \"0f84511edd9f7198fa05d8bc997d46b8b72d88df495481fc2a25bf0ddf4687a1\": rpc error: code = NotFound desc = could not find container \"0f84511edd9f7198fa05d8bc997d46b8b72d88df495481fc2a25bf0ddf4687a1\": container with ID starting with 0f84511edd9f7198fa05d8bc997d46b8b72d88df495481fc2a25bf0ddf4687a1 not found: ID does not exist"
Nov 02 12:38:53.720608 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:53.720575    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"]
Nov 02 12:38:53.771305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:53.771278    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-03c4e8-f89d5588f-5c4bq"]
Nov 02 12:38:55.580557 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:55.580487    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab65a1ba-c07c-4930-ac5b-e80eaadfe184" path="/var/lib/kubelet/pods/ab65a1ba-c07c-4930-ac5b-e80eaadfe184/volumes"
Nov 02 12:38:56.322490 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:56.322458    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67" podUID="27d181d3-3802-44f7-ace0-e66f40244776" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:38:58.188781 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.188746    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"]
Nov 02 12:38:58.189432 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.189411    4464 cpu_manager.go:414] "containerMap: removing container" podUID="ab65a1ba-c07c-4930-ac5b-e80eaadfe184" containerName="payload-processing"
Nov 02 12:38:58.189522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.189434    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab65a1ba-c07c-4930-ac5b-e80eaadfe184" containerName="payload-processing"
Nov 02 12:38:58.189522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.189447    4464 cpu_manager.go:414] "containerMap: removing container" podUID="07429504-309b-40e8-b1d5-9c7bde3b683d" containerName="revoke-keys"
Nov 02 12:38:58.189522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.189454    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07429504-309b-40e8-b1d5-9c7bde3b683d" containerName="revoke-keys"
Nov 02 12:38:58.189697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.189546    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab65a1ba-c07c-4930-ac5b-e80eaadfe184" containerName="payload-processing"
Nov 02 12:38:58.189697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.189562    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="07429504-309b-40e8-b1d5-9c7bde3b683d" containerName="revoke-keys"
Nov 02 12:38:58.189697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.189573    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d39d963-d570-44cc-96fd-a4efee1e45fc" containerName="istio-proxy"
Nov 02 12:38:58.193836 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.193816    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"
Nov 02 12:38:58.247620 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.247551    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:38:58.292060 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:38:58.292028    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:38:58.297910 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.297889    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:38:58.298029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.297928    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:38:58.298029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.297949    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:38:58.301146 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.301124    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"]
Nov 02 12:38:58.342337 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.342310    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j758k\" (UniqueName: \"kubernetes.io/projected/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-kube-api-access-j758k\") pod \"maas-api-revoke-sub-d6472f25-mkqn6\" (UID: \"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"
Nov 02 12:38:58.342446 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.342349    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6472f25-mkqn6\" (UID: \"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"
Nov 02 12:38:58.342509 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.342490    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6472f25-mkqn6\" (UID: \"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"
Nov 02 12:38:58.443632 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.443552    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6472f25-mkqn6\" (UID: \"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"
Nov 02 12:38:58.443632 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.443626    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j758k\" (UniqueName: \"kubernetes.io/projected/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-kube-api-access-j758k\") pod \"maas-api-revoke-sub-d6472f25-mkqn6\" (UID: \"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"
Nov 02 12:38:58.443840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.443656    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6472f25-mkqn6\" (UID: \"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"
Nov 02 12:38:58.444375 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.444351    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6472f25-mkqn6\" (UID: \"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"
Nov 02 12:38:58.446109 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.446086    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6472f25-mkqn6\" (UID: \"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"
Nov 02 12:38:58.507076 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.507046    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j758k\" (UniqueName: \"kubernetes.io/projected/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-kube-api-access-j758k\") pod \"maas-api-revoke-sub-d6472f25-mkqn6\" (UID: \"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"
Nov 02 12:38:58.806831 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:58.806801    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"
Nov 02 12:38:59.179199 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:59.179173    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"]
Nov 02 12:38:59.181058 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:38:59.181020    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a660cf_6e8c_4bca_aadb_5c99ff72c6b2.slice/crio-a7cc5af324ae3b16464dfe225e4a9d1467ff74bc83ff0c0a5dec86b41cefdb1d WatchSource:0}: Error finding container a7cc5af324ae3b16464dfe225e4a9d1467ff74bc83ff0c0a5dec86b41cefdb1d: Status 404 returned error can't find the container with id a7cc5af324ae3b16464dfe225e4a9d1467ff74bc83ff0c0a5dec86b41cefdb1d
Nov 02 12:38:59.371655 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:59.371628    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:38:59.601719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:59.601676    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6" event={"ID":"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2","Type":"ContainerStarted","Data":"d5cc73c125ecb9bcf0fc9e7a9cb6b0586764838d584d8576c14dbc29334b26dd"}
Nov 02 12:38:59.601935 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:59.601745    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6" event={"ID":"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2","Type":"ContainerStarted","Data":"a7cc5af324ae3b16464dfe225e4a9d1467ff74bc83ff0c0a5dec86b41cefdb1d"}
Nov 02 12:38:59.720398 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:38:59.720341    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6" podStartSLOduration=1.720327267 podStartE2EDuration="1.720327267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:59.71866744 +0000 UTC m=+2568.282375850" watchObservedRunningTime="2026-11-02 12:38:59.720327267 +0000 UTC m=+2568.284035675"
Nov 02 12:39:00.608016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:00.607965    4464 generic.go:356] "Generic (PLEG): container finished" podID="89a660cf-6e8c-4bca-aadb-5c99ff72c6b2" containerID="d5cc73c125ecb9bcf0fc9e7a9cb6b0586764838d584d8576c14dbc29334b26dd" exitCode=0
Nov 02 12:39:00.608415 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:00.608066    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6" event={"ID":"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2","Type":"ContainerDied","Data":"d5cc73c125ecb9bcf0fc9e7a9cb6b0586764838d584d8576c14dbc29334b26dd"}
Nov 02 12:39:01.743043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:01.743019    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"
Nov 02 12:39:01.783550 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:01.783527    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-kube-api-access-j758k\" (UniqueName: \"kubernetes.io/projected/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-kube-api-access-j758k\") pod \"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2\" (UID: \"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2\") "
Nov 02 12:39:01.783672 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:01.783574    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-maas-api-openshift-ca\") pod \"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2\" (UID: \"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2\") "
Nov 02 12:39:01.783960 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:01.783939    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-maas-api-openshift-ca" pod "89a660cf-6e8c-4bca-aadb-5c99ff72c6b2" (UID: "89a660cf-6e8c-4bca-aadb-5c99ff72c6b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:39:01.785428 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:01.785402    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-kube-api-access-j758k" pod "89a660cf-6e8c-4bca-aadb-5c99ff72c6b2" (UID: "89a660cf-6e8c-4bca-aadb-5c99ff72c6b2"). InnerVolumeSpecName "kube-api-access-j758k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:39:01.885147 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:01.885057    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-maas-api-xks-ca\") pod \"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2\" (UID: \"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2\") "
Nov 02 12:39:01.885399 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:01.885376    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j758k\" (UniqueName: \"kubernetes.io/projected/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-kube-api-access-j758k\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:01.885504 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:01.885402    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:01.886865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:01.886843    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-maas-api-xks-ca" pod "89a660cf-6e8c-4bca-aadb-5c99ff72c6b2" (UID: "89a660cf-6e8c-4bca-aadb-5c99ff72c6b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:39:01.986095 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:01.986070    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:02.619386 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:02.619352    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6" event={"ID":"89a660cf-6e8c-4bca-aadb-5c99ff72c6b2","Type":"ContainerDied","Data":"a7cc5af324ae3b16464dfe225e4a9d1467ff74bc83ff0c0a5dec86b41cefdb1d"}
Nov 02 12:39:02.619566 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:02.619399    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7cc5af324ae3b16464dfe225e4a9d1467ff74bc83ff0c0a5dec86b41cefdb1d"
Nov 02 12:39:02.619566 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:02.619361    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"
Nov 02 12:39:06.322860 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:06.322819    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67" podUID="27d181d3-3802-44f7-ace0-e66f40244776" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:39:11.255370 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.255290    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"]
Nov 02 12:39:11.255811 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.255796    4464 cpu_manager.go:414] "containerMap: removing container" podUID="89a660cf-6e8c-4bca-aadb-5c99ff72c6b2" containerName="revoke-keys"
Nov 02 12:39:11.255856 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.255813    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a660cf-6e8c-4bca-aadb-5c99ff72c6b2" containerName="revoke-keys"
Nov 02 12:39:11.255907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.255897    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a660cf-6e8c-4bca-aadb-5c99ff72c6b2" containerName="revoke-keys"
Nov 02 12:39:11.259057 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.259038    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"
Nov 02 12:39:11.313890 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.313866    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:39:11.364407 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.364375    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:39:11.367074 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.367048    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"]
Nov 02 12:39:11.369865 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.369842    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn4qt\" (UniqueName: \"kubernetes.io/projected/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-kube-api-access-jn4qt\") pod \"maas-api-revoke-sub-5207ddd4-ctdb7\" (UID: \"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"
Nov 02 12:39:11.370042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.370017    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5207ddd4-ctdb7\" (UID: \"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"
Nov 02 12:39:11.370084 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.370051    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5207ddd4-ctdb7\" (UID: \"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"
Nov 02 12:39:11.392540 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.392513    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:39:11.415078 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.415055    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:39:11.471323 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.471294    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5207ddd4-ctdb7\" (UID: \"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"
Nov 02 12:39:11.471475 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.471326    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5207ddd4-ctdb7\" (UID: \"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"
Nov 02 12:39:11.471475 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.471370    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4qt\" (UniqueName: \"kubernetes.io/projected/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-kube-api-access-jn4qt\") pod \"maas-api-revoke-sub-5207ddd4-ctdb7\" (UID: \"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"
Nov 02 12:39:11.472004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.471963    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5207ddd4-ctdb7\" (UID: \"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"
Nov 02 12:39:11.473740 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.473711    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5207ddd4-ctdb7\" (UID: \"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"
Nov 02 12:39:11.533365 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.533313    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn4qt\" (UniqueName: \"kubernetes.io/projected/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-kube-api-access-jn4qt\") pod \"maas-api-revoke-sub-5207ddd4-ctdb7\" (UID: \"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"
Nov 02 12:39:11.624095 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.624068    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:39:11.632241 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:11.632219    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"
Nov 02 12:39:12.003334 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:12.003307    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"]
Nov 02 12:39:12.004798 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:39:12.004771    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4fc0803_0ae8_4cf3_8b47_b6b1f838b089.slice/crio-ad50f2167d60507c95516b21524444aedf12c58495ae6fd4e94510a4ac9a594b WatchSource:0}: Error finding container ad50f2167d60507c95516b21524444aedf12c58495ae6fd4e94510a4ac9a594b: Status 404 returned error can't find the container with id ad50f2167d60507c95516b21524444aedf12c58495ae6fd4e94510a4ac9a594b
Nov 02 12:39:12.672799 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:12.672711    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7" event={"ID":"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089","Type":"ContainerStarted","Data":"0b7e4c1d18d8579b5938562198f800cbe1ca60a562e272e9034f1cc62d21d22e"}
Nov 02 12:39:12.672799 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:12.672765    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7" event={"ID":"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089","Type":"ContainerStarted","Data":"ad50f2167d60507c95516b21524444aedf12c58495ae6fd4e94510a4ac9a594b"}
Nov 02 12:39:12.792021 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:12.791947    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7" podStartSLOduration=1.791934614 podStartE2EDuration="1.791934614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:39:12.790280262 +0000 UTC m=+2581.353988674" watchObservedRunningTime="2026-11-02 12:39:12.791934614 +0000 UTC m=+2581.355643023"
Nov 02 12:39:13.317224 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.317190    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"]
Nov 02 12:39:13.318468 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.318445    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67" podUID="27d181d3-3802-44f7-ace0-e66f40244776" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:39:13.359917 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.359885    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"]
Nov 02 12:39:13.363543 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.363525    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"
Nov 02 12:39:13.429916 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.429883    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"]
Nov 02 12:39:13.489534 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.489504    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa74e18-0f22-40cc-ae3c-8a3290265806-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw\" (UID: \"2fa74e18-0f22-40cc-ae3c-8a3290265806\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"
Nov 02 12:39:13.489712 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.489557    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fa74e18-0f22-40cc-ae3c-8a3290265806-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw\" (UID: \"2fa74e18-0f22-40cc-ae3c-8a3290265806\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"
Nov 02 12:39:13.489712 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.489582    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvvsn\" (UniqueName: \"kubernetes.io/projected/2fa74e18-0f22-40cc-ae3c-8a3290265806-kube-api-access-gvvsn\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw\" (UID: \"2fa74e18-0f22-40cc-ae3c-8a3290265806\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"
Nov 02 12:39:13.489712 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.489670    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fa74e18-0f22-40cc-ae3c-8a3290265806-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw\" (UID: \"2fa74e18-0f22-40cc-ae3c-8a3290265806\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"
Nov 02 12:39:13.590693 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.590611    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa74e18-0f22-40cc-ae3c-8a3290265806-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw\" (UID: \"2fa74e18-0f22-40cc-ae3c-8a3290265806\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"
Nov 02 12:39:13.590693 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.590666    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fa74e18-0f22-40cc-ae3c-8a3290265806-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw\" (UID: \"2fa74e18-0f22-40cc-ae3c-8a3290265806\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"
Nov 02 12:39:13.590693 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.590691    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvvsn\" (UniqueName: \"kubernetes.io/projected/2fa74e18-0f22-40cc-ae3c-8a3290265806-kube-api-access-gvvsn\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw\" (UID: \"2fa74e18-0f22-40cc-ae3c-8a3290265806\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"
Nov 02 12:39:13.590921 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.590708    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fa74e18-0f22-40cc-ae3c-8a3290265806-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw\" (UID: \"2fa74e18-0f22-40cc-ae3c-8a3290265806\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"
Nov 02 12:39:13.591581 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.591552    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa74e18-0f22-40cc-ae3c-8a3290265806-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw\" (UID: \"2fa74e18-0f22-40cc-ae3c-8a3290265806\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"
Nov 02 12:39:13.591711 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.591666    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fa74e18-0f22-40cc-ae3c-8a3290265806-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw\" (UID: \"2fa74e18-0f22-40cc-ae3c-8a3290265806\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"
Nov 02 12:39:13.593194 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.593161    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fa74e18-0f22-40cc-ae3c-8a3290265806-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw\" (UID: \"2fa74e18-0f22-40cc-ae3c-8a3290265806\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"
Nov 02 12:39:13.652919 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.652886    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvvsn\" (UniqueName: \"kubernetes.io/projected/2fa74e18-0f22-40cc-ae3c-8a3290265806-kube-api-access-gvvsn\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw\" (UID: \"2fa74e18-0f22-40cc-ae3c-8a3290265806\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"
Nov 02 12:39:13.667244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.667217    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:39:13.676392 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.676368    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"
Nov 02 12:39:13.680226 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.680205    4464 generic.go:356] "Generic (PLEG): container finished" podID="f4fc0803-0ae8-4cf3-8b47-b6b1f838b089" containerID="0b7e4c1d18d8579b5938562198f800cbe1ca60a562e272e9034f1cc62d21d22e" exitCode=0
Nov 02 12:39:13.680329 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.680264    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7" event={"ID":"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089","Type":"ContainerDied","Data":"0b7e4c1d18d8579b5938562198f800cbe1ca60a562e272e9034f1cc62d21d22e"}
Nov 02 12:39:13.813968 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.813931    4464 scope.go:122] "RemoveContainer" containerID="5587c946fad483f440f7f4ccc94254ed2a658671f453c373d06e24b90fbd7b7c"
Nov 02 12:39:13.825546 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:13.825522    4464 scope.go:122] "RemoveContainer" containerID="51073f63168072b4161f1eca3cfdd26735e9d97135d598628c5ac20dd5ea3e96"
Nov 02 12:39:14.050162 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:14.050130    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"]
Nov 02 12:39:14.051494 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:39:14.051459    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa74e18_0f22_40cc_ae3c_8a3290265806.slice/crio-874d7a50e92b487cfd9279591edd315833d7c17f4c7f1e5ae4eeafdf5668c184 WatchSource:0}: Error finding container 874d7a50e92b487cfd9279591edd315833d7c17f4c7f1e5ae4eeafdf5668c184: Status 404 returned error can't find the container with id 874d7a50e92b487cfd9279591edd315833d7c17f4c7f1e5ae4eeafdf5668c184
Nov 02 12:39:14.687530 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:14.687494    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw" event={"ID":"2fa74e18-0f22-40cc-ae3c-8a3290265806","Type":"ContainerStarted","Data":"e50586aa205971eb208037648eee62deaaf72af89fa3a4df8ef2ebbcb075b719"}
Nov 02 12:39:14.687965 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:14.687759    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"
Nov 02 12:39:14.687965 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:14.687818    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw" event={"ID":"2fa74e18-0f22-40cc-ae3c-8a3290265806","Type":"ContainerStarted","Data":"874d7a50e92b487cfd9279591edd315833d7c17f4c7f1e5ae4eeafdf5668c184"}
Nov 02 12:39:14.808575 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:14.808528    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw" podStartSLOduration=1.8085155149999999 podStartE2EDuration="1.808515515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:39:14.807184645 +0000 UTC m=+2583.370893055" watchObservedRunningTime="2026-11-02 12:39:14.808515515 +0000 UTC m=+2583.372223924"
Nov 02 12:39:14.818363 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:14.818342    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"
Nov 02 12:39:14.903242 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:14.903213    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-kube-api-access-jn4qt\" (UniqueName: \"kubernetes.io/projected/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-kube-api-access-jn4qt\") pod \"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089\" (UID: \"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089\") "
Nov 02 12:39:14.903414 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:14.903307    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-maas-api-xks-ca\") pod \"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089\" (UID: \"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089\") "
Nov 02 12:39:14.903414 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:14.903403    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-maas-api-openshift-ca\") pod \"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089\" (UID: \"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089\") "
Nov 02 12:39:14.903714 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:14.903692    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-maas-api-openshift-ca" pod "f4fc0803-0ae8-4cf3-8b47-b6b1f838b089" (UID: "f4fc0803-0ae8-4cf3-8b47-b6b1f838b089"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:39:14.905194 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:14.905174    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-kube-api-access-jn4qt" pod "f4fc0803-0ae8-4cf3-8b47-b6b1f838b089" (UID: "f4fc0803-0ae8-4cf3-8b47-b6b1f838b089"). InnerVolumeSpecName "kube-api-access-jn4qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:39:14.905313 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:14.905294    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-maas-api-xks-ca" pod "f4fc0803-0ae8-4cf3-8b47-b6b1f838b089" (UID: "f4fc0803-0ae8-4cf3-8b47-b6b1f838b089"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:39:15.004357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:15.004329    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:15.004357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:15.004356    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:15.004564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:15.004368    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn4qt\" (UniqueName: \"kubernetes.io/projected/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089-kube-api-access-jn4qt\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:15.693715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:15.693679    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"
Nov 02 12:39:15.693715 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:15.693699    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7" event={"ID":"f4fc0803-0ae8-4cf3-8b47-b6b1f838b089","Type":"ContainerDied","Data":"ad50f2167d60507c95516b21524444aedf12c58495ae6fd4e94510a4ac9a594b"}
Nov 02 12:39:15.694166 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:15.693744    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad50f2167d60507c95516b21524444aedf12c58495ae6fd4e94510a4ac9a594b"
Nov 02 12:39:18.317651 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.317613    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67" podUID="27d181d3-3802-44f7-ace0-e66f40244776" containerName="payload-processing" containerID="cri-o://c29ca4587fe88f78d2eed57a99c6d349d91e1b55cd9735d95ae57f165c4e38d6" gracePeriod=25
Nov 02 12:39:18.578344 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.578284    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"
Nov 02 12:39:18.635966 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.635933    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d181d3-3802-44f7-ace0-e66f40244776-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d181d3-3802-44f7-ace0-e66f40244776-plugins-config-volume\") pod \"27d181d3-3802-44f7-ace0-e66f40244776\" (UID: \"27d181d3-3802-44f7-ace0-e66f40244776\") "
Nov 02 12:39:18.636365 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.636087    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d181d3-3802-44f7-ace0-e66f40244776-provider-credentials\" (UniqueName: \"kubernetes.io/projected/27d181d3-3802-44f7-ace0-e66f40244776-provider-credentials\") pod \"27d181d3-3802-44f7-ace0-e66f40244776\" (UID: \"27d181d3-3802-44f7-ace0-e66f40244776\") "
Nov 02 12:39:18.636365 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.636198    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d181d3-3802-44f7-ace0-e66f40244776-kube-api-access-nqlgt\" (UniqueName: \"kubernetes.io/projected/27d181d3-3802-44f7-ace0-e66f40244776-kube-api-access-nqlgt\") pod \"27d181d3-3802-44f7-ace0-e66f40244776\" (UID: \"27d181d3-3802-44f7-ace0-e66f40244776\") "
Nov 02 12:39:18.636365 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.636227    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d181d3-3802-44f7-ace0-e66f40244776-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27d181d3-3802-44f7-ace0-e66f40244776-routing-overlay-volume\") pod \"27d181d3-3802-44f7-ace0-e66f40244776\" (UID: \"27d181d3-3802-44f7-ace0-e66f40244776\") "
Nov 02 12:39:18.636647 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.636469    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d181d3-3802-44f7-ace0-e66f40244776-plugins-config-volume" pod "27d181d3-3802-44f7-ace0-e66f40244776" (UID: "27d181d3-3802-44f7-ace0-e66f40244776"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:39:18.636790 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.636769    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27d181d3-3802-44f7-ace0-e66f40244776-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:18.637028 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.637006    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d181d3-3802-44f7-ace0-e66f40244776-routing-overlay-volume" pod "27d181d3-3802-44f7-ace0-e66f40244776" (UID: "27d181d3-3802-44f7-ace0-e66f40244776"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:39:18.638473 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.638447    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d181d3-3802-44f7-ace0-e66f40244776-provider-credentials" pod "27d181d3-3802-44f7-ace0-e66f40244776" (UID: "27d181d3-3802-44f7-ace0-e66f40244776"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:39:18.638564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.638486    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d181d3-3802-44f7-ace0-e66f40244776-kube-api-access-nqlgt" pod "27d181d3-3802-44f7-ace0-e66f40244776" (UID: "27d181d3-3802-44f7-ace0-e66f40244776"). InnerVolumeSpecName "kube-api-access-nqlgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:39:18.711539 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.711506    4464 generic.go:356] "Generic (PLEG): container finished" podID="27d181d3-3802-44f7-ace0-e66f40244776" containerID="c29ca4587fe88f78d2eed57a99c6d349d91e1b55cd9735d95ae57f165c4e38d6" exitCode=0
Nov 02 12:39:18.711719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.711548    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"
Nov 02 12:39:18.711719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.711573    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67" event={"ID":"27d181d3-3802-44f7-ace0-e66f40244776","Type":"ContainerDied","Data":"c29ca4587fe88f78d2eed57a99c6d349d91e1b55cd9735d95ae57f165c4e38d6"}
Nov 02 12:39:18.711719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.711616    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67" event={"ID":"27d181d3-3802-44f7-ace0-e66f40244776","Type":"ContainerDied","Data":"abc7c7e762077026afb567136b7fb0019d3c65f172abde7dacd5e7164eb6b8c0"}
Nov 02 12:39:18.711719 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.711638    4464 scope.go:122] "RemoveContainer" containerID="c29ca4587fe88f78d2eed57a99c6d349d91e1b55cd9735d95ae57f165c4e38d6"
Nov 02 12:39:18.722504 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.722488    4464 scope.go:122] "RemoveContainer" containerID="c29ca4587fe88f78d2eed57a99c6d349d91e1b55cd9735d95ae57f165c4e38d6"
Nov 02 12:39:18.722768 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:39:18.722749    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c29ca4587fe88f78d2eed57a99c6d349d91e1b55cd9735d95ae57f165c4e38d6\": container with ID starting with c29ca4587fe88f78d2eed57a99c6d349d91e1b55cd9735d95ae57f165c4e38d6 not found: ID does not exist" containerID="c29ca4587fe88f78d2eed57a99c6d349d91e1b55cd9735d95ae57f165c4e38d6"
Nov 02 12:39:18.722827 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.722774    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c29ca4587fe88f78d2eed57a99c6d349d91e1b55cd9735d95ae57f165c4e38d6"} err="failed to get container status \"c29ca4587fe88f78d2eed57a99c6d349d91e1b55cd9735d95ae57f165c4e38d6\": rpc error: code = NotFound desc = could not find container \"c29ca4587fe88f78d2eed57a99c6d349d91e1b55cd9735d95ae57f165c4e38d6\": container with ID starting with c29ca4587fe88f78d2eed57a99c6d349d91e1b55cd9735d95ae57f165c4e38d6 not found: ID does not exist"
Nov 02 12:39:18.737451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.737430    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqlgt\" (UniqueName: \"kubernetes.io/projected/27d181d3-3802-44f7-ace0-e66f40244776-kube-api-access-nqlgt\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:18.737451 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.737450    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27d181d3-3802-44f7-ace0-e66f40244776-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:18.737570 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.737465    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27d181d3-3802-44f7-ace0-e66f40244776-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:18.879490 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.879403    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"]
Nov 02 12:39:18.883607 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:18.883576    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696twg67"]
Nov 02 12:39:19.581125 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:19.581094    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d181d3-3802-44f7-ace0-e66f40244776" path="/var/lib/kubelet/pods/27d181d3-3802-44f7-ace0-e66f40244776/volumes"
Nov 02 12:39:24.175258 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.175223    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"]
Nov 02 12:39:24.175700 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.175687    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f4fc0803-0ae8-4cf3-8b47-b6b1f838b089" containerName="revoke-keys"
Nov 02 12:39:24.175747 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.175701    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4fc0803-0ae8-4cf3-8b47-b6b1f838b089" containerName="revoke-keys"
Nov 02 12:39:24.175747 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.175718    4464 cpu_manager.go:414] "containerMap: removing container" podUID="27d181d3-3802-44f7-ace0-e66f40244776" containerName="payload-processing"
Nov 02 12:39:24.175747 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.175724    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d181d3-3802-44f7-ace0-e66f40244776" containerName="payload-processing"
Nov 02 12:39:24.175845 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.175787    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4fc0803-0ae8-4cf3-8b47-b6b1f838b089" containerName="revoke-keys"
Nov 02 12:39:24.175845 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.175798    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d181d3-3802-44f7-ace0-e66f40244776" containerName="payload-processing"
Nov 02 12:39:24.180316 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.180296    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"
Nov 02 12:39:24.284312 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.284275    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:39:24.284515 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.284322    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:39:24.284515 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.284448    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:39:24.284609 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.284516    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:39:24.286924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.286881    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"]
Nov 02 12:39:24.291037 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.291014    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00497b1b-8136-453b-a1a9-2b136c39ba67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d757018-gstjm\" (UID: \"00497b1b-8136-453b-a1a9-2b136c39ba67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"
Nov 02 12:39:24.291175 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.291154    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfjr6\" (UniqueName: \"kubernetes.io/projected/00497b1b-8136-453b-a1a9-2b136c39ba67-kube-api-access-vfjr6\") pod \"maas-api-revoke-sub-3d757018-gstjm\" (UID: \"00497b1b-8136-453b-a1a9-2b136c39ba67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"
Nov 02 12:39:24.291225 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.291191    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00497b1b-8136-453b-a1a9-2b136c39ba67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d757018-gstjm\" (UID: \"00497b1b-8136-453b-a1a9-2b136c39ba67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"
Nov 02 12:39:24.391820 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.391788    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfjr6\" (UniqueName: \"kubernetes.io/projected/00497b1b-8136-453b-a1a9-2b136c39ba67-kube-api-access-vfjr6\") pod \"maas-api-revoke-sub-3d757018-gstjm\" (UID: \"00497b1b-8136-453b-a1a9-2b136c39ba67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"
Nov 02 12:39:24.392025 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.391828    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00497b1b-8136-453b-a1a9-2b136c39ba67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d757018-gstjm\" (UID: \"00497b1b-8136-453b-a1a9-2b136c39ba67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"
Nov 02 12:39:24.392025 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.391877    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00497b1b-8136-453b-a1a9-2b136c39ba67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d757018-gstjm\" (UID: \"00497b1b-8136-453b-a1a9-2b136c39ba67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"
Nov 02 12:39:24.392497 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.392473    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00497b1b-8136-453b-a1a9-2b136c39ba67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d757018-gstjm\" (UID: \"00497b1b-8136-453b-a1a9-2b136c39ba67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"
Nov 02 12:39:24.394163 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.394142    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00497b1b-8136-453b-a1a9-2b136c39ba67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d757018-gstjm\" (UID: \"00497b1b-8136-453b-a1a9-2b136c39ba67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"
Nov 02 12:39:24.454239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.454169    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfjr6\" (UniqueName: \"kubernetes.io/projected/00497b1b-8136-453b-a1a9-2b136c39ba67-kube-api-access-vfjr6\") pod \"maas-api-revoke-sub-3d757018-gstjm\" (UID: \"00497b1b-8136-453b-a1a9-2b136c39ba67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"
Nov 02 12:39:24.492969 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.492935    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"
Nov 02 12:39:24.562760 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.562732    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"]
Nov 02 12:39:24.565548 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.565515    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw" podUID="2fa74e18-0f22-40cc-ae3c-8a3290265806" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:39:24.598010 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.597968    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"]
Nov 02 12:39:24.603139 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.603115    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"
Nov 02 12:39:24.659767 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.659732    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"]
Nov 02 12:39:24.695391 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.695359    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w\" (UID: \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"
Nov 02 12:39:24.695597 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.695427    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w\" (UID: \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"
Nov 02 12:39:24.695597 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.695475    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdb86\" (UniqueName: \"kubernetes.io/projected/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-kube-api-access-tdb86\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w\" (UID: \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"
Nov 02 12:39:24.695597 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.695578    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w\" (UID: \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"
Nov 02 12:39:24.804126 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.804084    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w\" (UID: \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"
Nov 02 12:39:24.804345 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.804143    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdb86\" (UniqueName: \"kubernetes.io/projected/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-kube-api-access-tdb86\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w\" (UID: \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"
Nov 02 12:39:24.804345 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.804233    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w\" (UID: \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"
Nov 02 12:39:24.804345 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.804307    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w\" (UID: \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"
Nov 02 12:39:24.805185 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.805126    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w\" (UID: \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"
Nov 02 12:39:24.805604 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.805578    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w\" (UID: \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"
Nov 02 12:39:24.807771 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.807693    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w\" (UID: \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"
Nov 02 12:39:24.865610 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.865584    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"]
Nov 02 12:39:24.866780 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.866753    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdb86\" (UniqueName: \"kubernetes.io/projected/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-kube-api-access-tdb86\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w\" (UID: \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"
Nov 02 12:39:24.866939 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:39:24.866914    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00497b1b_8136_453b_a1a9_2b136c39ba67.slice/crio-67422236dd89d501847e2940408762b9687694715ce2b5fec12ac81bd0f8244f WatchSource:0}: Error finding container 67422236dd89d501847e2940408762b9687694715ce2b5fec12ac81bd0f8244f: Status 404 returned error can't find the container with id 67422236dd89d501847e2940408762b9687694715ce2b5fec12ac81bd0f8244f
Nov 02 12:39:24.917206 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:24.917184    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"
Nov 02 12:39:25.290213 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:25.290190    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"]
Nov 02 12:39:25.291764 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:39:25.291738    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecfe8cd_93c7_4e1b_9f22_e1f59ac64fbd.slice/crio-fde5ab1cae3047d75a884dc97a67eecadb4a4832a8cea2ec3b6a0510fdd9d68f WatchSource:0}: Error finding container fde5ab1cae3047d75a884dc97a67eecadb4a4832a8cea2ec3b6a0510fdd9d68f: Status 404 returned error can't find the container with id fde5ab1cae3047d75a884dc97a67eecadb4a4832a8cea2ec3b6a0510fdd9d68f
Nov 02 12:39:25.755258 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:25.755209    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w" event={"ID":"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd","Type":"ContainerStarted","Data":"76bfc8737fe97fa84d502f88ee06c63c78b68fbf60d539ea2087837cc4cfc94a"}
Nov 02 12:39:25.755449 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:25.755316    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w" event={"ID":"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd","Type":"ContainerStarted","Data":"fde5ab1cae3047d75a884dc97a67eecadb4a4832a8cea2ec3b6a0510fdd9d68f"}
Nov 02 12:39:25.755449 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:25.755350    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"
Nov 02 12:39:25.756804 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:25.756776    4464 generic.go:356] "Generic (PLEG): container finished" podID="00497b1b-8136-453b-a1a9-2b136c39ba67" containerID="309ec739bd4839cdec6953e4c66f68c463236f59e3f0d04e649ff51ebc28ebd6" exitCode=0
Nov 02 12:39:25.756909 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:25.756844    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm" event={"ID":"00497b1b-8136-453b-a1a9-2b136c39ba67","Type":"ContainerDied","Data":"309ec739bd4839cdec6953e4c66f68c463236f59e3f0d04e649ff51ebc28ebd6"}
Nov 02 12:39:25.756909 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:25.756875    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm" event={"ID":"00497b1b-8136-453b-a1a9-2b136c39ba67","Type":"ContainerStarted","Data":"67422236dd89d501847e2940408762b9687694715ce2b5fec12ac81bd0f8244f"}
Nov 02 12:39:25.874676 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:25.874605    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w" podStartSLOduration=1.874590726 podStartE2EDuration="1.874590726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:39:25.873501551 +0000 UTC m=+2594.437209960" watchObservedRunningTime="2026-11-02 12:39:25.874590726 +0000 UTC m=+2594.438299134"
Nov 02 12:39:25.963902 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:25.963865    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:39:26.894113 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:26.894090    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"
Nov 02 12:39:27.025811 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:27.025734    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00497b1b-8136-453b-a1a9-2b136c39ba67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00497b1b-8136-453b-a1a9-2b136c39ba67-maas-api-openshift-ca\") pod \"00497b1b-8136-453b-a1a9-2b136c39ba67\" (UID: \"00497b1b-8136-453b-a1a9-2b136c39ba67\") "
Nov 02 12:39:27.025811 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:27.025784    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00497b1b-8136-453b-a1a9-2b136c39ba67-kube-api-access-vfjr6\" (UniqueName: \"kubernetes.io/projected/00497b1b-8136-453b-a1a9-2b136c39ba67-kube-api-access-vfjr6\") pod \"00497b1b-8136-453b-a1a9-2b136c39ba67\" (UID: \"00497b1b-8136-453b-a1a9-2b136c39ba67\") "
Nov 02 12:39:27.026083 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:27.025865    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00497b1b-8136-453b-a1a9-2b136c39ba67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00497b1b-8136-453b-a1a9-2b136c39ba67-maas-api-xks-ca\") pod \"00497b1b-8136-453b-a1a9-2b136c39ba67\" (UID: \"00497b1b-8136-453b-a1a9-2b136c39ba67\") "
Nov 02 12:39:27.026167 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:27.026137    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00497b1b-8136-453b-a1a9-2b136c39ba67-maas-api-openshift-ca" pod "00497b1b-8136-453b-a1a9-2b136c39ba67" (UID: "00497b1b-8136-453b-a1a9-2b136c39ba67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:39:27.028127 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:27.028097    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00497b1b-8136-453b-a1a9-2b136c39ba67-maas-api-xks-ca" pod "00497b1b-8136-453b-a1a9-2b136c39ba67" (UID: "00497b1b-8136-453b-a1a9-2b136c39ba67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:39:27.028260 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:27.028161    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00497b1b-8136-453b-a1a9-2b136c39ba67-kube-api-access-vfjr6" pod "00497b1b-8136-453b-a1a9-2b136c39ba67" (UID: "00497b1b-8136-453b-a1a9-2b136c39ba67"). InnerVolumeSpecName "kube-api-access-vfjr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:39:27.126653 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:27.126618    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00497b1b-8136-453b-a1a9-2b136c39ba67-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:27.126653 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:27.126651    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00497b1b-8136-453b-a1a9-2b136c39ba67-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:27.126803 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:27.126665    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfjr6\" (UniqueName: \"kubernetes.io/projected/00497b1b-8136-453b-a1a9-2b136c39ba67-kube-api-access-vfjr6\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:27.768221 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:27.768183    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm" event={"ID":"00497b1b-8136-453b-a1a9-2b136c39ba67","Type":"ContainerDied","Data":"67422236dd89d501847e2940408762b9687694715ce2b5fec12ac81bd0f8244f"}
Nov 02 12:39:27.768388 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:27.768232    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67422236dd89d501847e2940408762b9687694715ce2b5fec12ac81bd0f8244f"
Nov 02 12:39:27.768388 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:27.768199    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"
Nov 02 12:39:28.292851 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:39:28.292817    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:39:29.563233 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:29.563181    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw" podUID="2fa74e18-0f22-40cc-ae3c-8a3290265806" containerName="payload-processing" containerID="cri-o://e50586aa205971eb208037648eee62deaaf72af89fa3a4df8ef2ebbcb075b719" gracePeriod=25
Nov 02 12:39:29.779725 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:29.779688    4464 generic.go:356] "Generic (PLEG): container finished" podID="2fa74e18-0f22-40cc-ae3c-8a3290265806" containerID="e50586aa205971eb208037648eee62deaaf72af89fa3a4df8ef2ebbcb075b719" exitCode=0
Nov 02 12:39:29.779924 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:29.779754    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw" event={"ID":"2fa74e18-0f22-40cc-ae3c-8a3290265806","Type":"ContainerDied","Data":"e50586aa205971eb208037648eee62deaaf72af89fa3a4df8ef2ebbcb075b719"}
Nov 02 12:39:29.812563 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:29.812540    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"
Nov 02 12:39:29.954299 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:29.954218    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa74e18-0f22-40cc-ae3c-8a3290265806-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa74e18-0f22-40cc-ae3c-8a3290265806-plugins-config-volume\") pod \"2fa74e18-0f22-40cc-ae3c-8a3290265806\" (UID: \"2fa74e18-0f22-40cc-ae3c-8a3290265806\") "
Nov 02 12:39:29.954442 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:29.954409    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa74e18-0f22-40cc-ae3c-8a3290265806-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fa74e18-0f22-40cc-ae3c-8a3290265806-provider-credentials\") pod \"2fa74e18-0f22-40cc-ae3c-8a3290265806\" (UID: \"2fa74e18-0f22-40cc-ae3c-8a3290265806\") "
Nov 02 12:39:29.954442 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:29.954439    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa74e18-0f22-40cc-ae3c-8a3290265806-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fa74e18-0f22-40cc-ae3c-8a3290265806-routing-overlay-volume\") pod \"2fa74e18-0f22-40cc-ae3c-8a3290265806\" (UID: \"2fa74e18-0f22-40cc-ae3c-8a3290265806\") "
Nov 02 12:39:29.954542 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:29.954461    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa74e18-0f22-40cc-ae3c-8a3290265806-kube-api-access-gvvsn\" (UniqueName: \"kubernetes.io/projected/2fa74e18-0f22-40cc-ae3c-8a3290265806-kube-api-access-gvvsn\") pod \"2fa74e18-0f22-40cc-ae3c-8a3290265806\" (UID: \"2fa74e18-0f22-40cc-ae3c-8a3290265806\") "
Nov 02 12:39:29.954778 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:29.954750    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa74e18-0f22-40cc-ae3c-8a3290265806-plugins-config-volume" pod "2fa74e18-0f22-40cc-ae3c-8a3290265806" (UID: "2fa74e18-0f22-40cc-ae3c-8a3290265806"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:39:29.954831 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:29.954808    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa74e18-0f22-40cc-ae3c-8a3290265806-routing-overlay-volume" pod "2fa74e18-0f22-40cc-ae3c-8a3290265806" (UID: "2fa74e18-0f22-40cc-ae3c-8a3290265806"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:39:29.956616 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:29.956588    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa74e18-0f22-40cc-ae3c-8a3290265806-kube-api-access-gvvsn" pod "2fa74e18-0f22-40cc-ae3c-8a3290265806" (UID: "2fa74e18-0f22-40cc-ae3c-8a3290265806"). InnerVolumeSpecName "kube-api-access-gvvsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:39:29.956616 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:29.956609    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa74e18-0f22-40cc-ae3c-8a3290265806-provider-credentials" pod "2fa74e18-0f22-40cc-ae3c-8a3290265806" (UID: "2fa74e18-0f22-40cc-ae3c-8a3290265806"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:39:30.055446 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:30.055413    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa74e18-0f22-40cc-ae3c-8a3290265806-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:30.055446 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:30.055441    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2fa74e18-0f22-40cc-ae3c-8a3290265806-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:30.055446 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:30.055451    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2fa74e18-0f22-40cc-ae3c-8a3290265806-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:30.055679 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:30.055459    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvvsn\" (UniqueName: \"kubernetes.io/projected/2fa74e18-0f22-40cc-ae3c-8a3290265806-kube-api-access-gvvsn\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:30.786826 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:30.786794    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"
Nov 02 12:39:30.786826 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:30.786810    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw" event={"ID":"2fa74e18-0f22-40cc-ae3c-8a3290265806","Type":"ContainerDied","Data":"874d7a50e92b487cfd9279591edd315833d7c17f4c7f1e5ae4eeafdf5668c184"}
Nov 02 12:39:30.787341 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:30.786864    4464 scope.go:122] "RemoveContainer" containerID="e50586aa205971eb208037648eee62deaaf72af89fa3a4df8ef2ebbcb075b719"
Nov 02 12:39:30.953705 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:30.953670    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"]
Nov 02 12:39:31.004539 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:31.004510    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-69b9hfctw"]
Nov 02 12:39:31.581150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:31.581116    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa74e18-0f22-40cc-ae3c-8a3290265806" path="/var/lib/kubelet/pods/2fa74e18-0f22-40cc-ae3c-8a3290265806/volumes"
Nov 02 12:39:36.465012 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:36.464968    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"]
Nov 02 12:39:36.467533 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:36.467500    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w" podUID="decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:39:39.478563 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:39.478530    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:39:41.465995 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.465918    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w" podUID="decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd" containerName="payload-processing" containerID="cri-o://76bfc8737fe97fa84d502f88ee06c63c78b68fbf60d539ea2087837cc4cfc94a" gracePeriod=25
Nov 02 12:39:41.713465 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.713438    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"
Nov 02 12:39:41.756772 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.756749    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-provider-credentials\") pod \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\" (UID: \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\") "
Nov 02 12:39:41.756917 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.756869    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-routing-overlay-volume\") pod \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\" (UID: \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\") "
Nov 02 12:39:41.757013 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.756941    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-plugins-config-volume\") pod \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\" (UID: \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\") "
Nov 02 12:39:41.757255 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.757227    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-routing-overlay-volume" pod "decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd" (UID: "decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:39:41.757364 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.757330    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-plugins-config-volume" pod "decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd" (UID: "decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:39:41.758764 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.758744    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-provider-credentials" pod "decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd" (UID: "decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:39:41.844452 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.844418    4464 generic.go:356] "Generic (PLEG): container finished" podID="decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd" containerID="76bfc8737fe97fa84d502f88ee06c63c78b68fbf60d539ea2087837cc4cfc94a" exitCode=0
Nov 02 12:39:41.844632 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.844462    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"
Nov 02 12:39:41.844632 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.844510    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w" event={"ID":"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd","Type":"ContainerDied","Data":"76bfc8737fe97fa84d502f88ee06c63c78b68fbf60d539ea2087837cc4cfc94a"}
Nov 02 12:39:41.844632 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.844558    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w" event={"ID":"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd","Type":"ContainerDied","Data":"fde5ab1cae3047d75a884dc97a67eecadb4a4832a8cea2ec3b6a0510fdd9d68f"}
Nov 02 12:39:41.844632 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.844581    4464 scope.go:122] "RemoveContainer" containerID="76bfc8737fe97fa84d502f88ee06c63c78b68fbf60d539ea2087837cc4cfc94a"
Nov 02 12:39:41.853122 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.853066    4464 scope.go:122] "RemoveContainer" containerID="76bfc8737fe97fa84d502f88ee06c63c78b68fbf60d539ea2087837cc4cfc94a"
Nov 02 12:39:41.853358 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:39:41.853339    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76bfc8737fe97fa84d502f88ee06c63c78b68fbf60d539ea2087837cc4cfc94a\": container with ID starting with 76bfc8737fe97fa84d502f88ee06c63c78b68fbf60d539ea2087837cc4cfc94a not found: ID does not exist" containerID="76bfc8737fe97fa84d502f88ee06c63c78b68fbf60d539ea2087837cc4cfc94a"
Nov 02 12:39:41.853403 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.853365    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76bfc8737fe97fa84d502f88ee06c63c78b68fbf60d539ea2087837cc4cfc94a"} err="failed to get container status \"76bfc8737fe97fa84d502f88ee06c63c78b68fbf60d539ea2087837cc4cfc94a\": rpc error: code = NotFound desc = could not find container \"76bfc8737fe97fa84d502f88ee06c63c78b68fbf60d539ea2087837cc4cfc94a\": container with ID starting with 76bfc8737fe97fa84d502f88ee06c63c78b68fbf60d539ea2087837cc4cfc94a not found: ID does not exist"
Nov 02 12:39:41.857657 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.857638    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-kube-api-access-tdb86\" (UniqueName: \"kubernetes.io/projected/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-kube-api-access-tdb86\") pod \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\" (UID: \"decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd\") "
Nov 02 12:39:41.857895 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.857883    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:41.857945 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.857897    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:41.857945 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.857906    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:41.859471 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.859452    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-kube-api-access-tdb86" pod "decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd" (UID: "decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd"). InnerVolumeSpecName "kube-api-access-tdb86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:39:41.959206 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:41.959172    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdb86\" (UniqueName: \"kubernetes.io/projected/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd-kube-api-access-tdb86\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:39:42.266725 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:42.266694    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"]
Nov 02 12:39:42.317121 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:42.317090    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-54d8jbr9w"]
Nov 02 12:39:43.580722 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:43.580685    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd" path="/var/lib/kubelet/pods/decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd/volumes"
Nov 02 12:39:54.874071 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.874028    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"]
Nov 02 12:39:54.874733 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.874712    4464 cpu_manager.go:414] "containerMap: removing container" podUID="decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd" containerName="payload-processing"
Nov 02 12:39:54.874840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.874734    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd" containerName="payload-processing"
Nov 02 12:39:54.874840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.874762    4464 cpu_manager.go:414] "containerMap: removing container" podUID="2fa74e18-0f22-40cc-ae3c-8a3290265806" containerName="payload-processing"
Nov 02 12:39:54.874840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.874770    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa74e18-0f22-40cc-ae3c-8a3290265806" containerName="payload-processing"
Nov 02 12:39:54.874840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.874796    4464 cpu_manager.go:414] "containerMap: removing container" podUID="00497b1b-8136-453b-a1a9-2b136c39ba67" containerName="revoke-keys"
Nov 02 12:39:54.874840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.874803    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00497b1b-8136-453b-a1a9-2b136c39ba67" containerName="revoke-keys"
Nov 02 12:39:54.875134 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.874910    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa74e18-0f22-40cc-ae3c-8a3290265806" containerName="payload-processing"
Nov 02 12:39:54.875134 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.874926    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="00497b1b-8136-453b-a1a9-2b136c39ba67" containerName="revoke-keys"
Nov 02 12:39:54.875134 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.874943    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="decfe8cd-93c7-4e1b-9f22-e1f59ac64fbd" containerName="payload-processing"
Nov 02 12:39:54.879527 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.879506    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:39:54.981915 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.981877    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c8f58de-36cb-403f-ba87-965c05827381-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7\" (UID: \"8c8f58de-36cb-403f-ba87-965c05827381\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:39:54.981915 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.981919    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhrgd\" (UniqueName: \"kubernetes.io/projected/8c8f58de-36cb-403f-ba87-965c05827381-kube-api-access-hhrgd\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7\" (UID: \"8c8f58de-36cb-403f-ba87-965c05827381\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:39:54.982152 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.981953    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c8f58de-36cb-403f-ba87-965c05827381-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7\" (UID: \"8c8f58de-36cb-403f-ba87-965c05827381\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:39:54.982152 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.982061    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8f58de-36cb-403f-ba87-965c05827381-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7\" (UID: \"8c8f58de-36cb-403f-ba87-965c05827381\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:39:54.983534 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.983478    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"routing-overlay\""
Nov 02 12:39:54.983534 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.983478    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"openshift-service-ca.crt\""
Nov 02 12:39:54.983534 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.983515    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"kube-root-ca.crt\""
Nov 02 12:39:54.983534 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.983530    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"payload-processing-external-model-e2e--75a83f02-dockercfg-gj6qx\""
Nov 02 12:39:54.983784 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.983557    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"e2e-cmp-legacy-e3b4b0-provider-cred\""
Nov 02 12:39:54.983784 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.983563    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6ac03f\""
Nov 02 12:39:54.985872 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:54.985849    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"]
Nov 02 12:39:55.083096 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:55.083070    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c8f58de-36cb-403f-ba87-965c05827381-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7\" (UID: \"8c8f58de-36cb-403f-ba87-965c05827381\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:39:55.083248 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:55.083100    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhrgd\" (UniqueName: \"kubernetes.io/projected/8c8f58de-36cb-403f-ba87-965c05827381-kube-api-access-hhrgd\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7\" (UID: \"8c8f58de-36cb-403f-ba87-965c05827381\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:39:55.083248 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:55.083131    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c8f58de-36cb-403f-ba87-965c05827381-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7\" (UID: \"8c8f58de-36cb-403f-ba87-965c05827381\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:39:55.083378 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:55.083256    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8f58de-36cb-403f-ba87-965c05827381-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7\" (UID: \"8c8f58de-36cb-403f-ba87-965c05827381\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:39:55.083796 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:55.083766    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c8f58de-36cb-403f-ba87-965c05827381-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7\" (UID: \"8c8f58de-36cb-403f-ba87-965c05827381\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:39:55.084017 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:55.083971    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8f58de-36cb-403f-ba87-965c05827381-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7\" (UID: \"8c8f58de-36cb-403f-ba87-965c05827381\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:39:55.085535 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:55.085517    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c8f58de-36cb-403f-ba87-965c05827381-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7\" (UID: \"8c8f58de-36cb-403f-ba87-965c05827381\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:39:55.145179 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:55.145126    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhrgd\" (UniqueName: \"kubernetes.io/projected/8c8f58de-36cb-403f-ba87-965c05827381-kube-api-access-hhrgd\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7\" (UID: \"8c8f58de-36cb-403f-ba87-965c05827381\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:39:55.194239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:55.194216    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:39:55.390595 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:55.390562    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:39:55.566624 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:55.566583    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"]
Nov 02 12:39:55.567912 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:39:55.567881    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8f58de_36cb_403f_ba87_965c05827381.slice/crio-9c2e6b21246927ca4a9f3ecd01a84051350279815ead1b40b04e306bc1054692 WatchSource:0}: Error finding container 9c2e6b21246927ca4a9f3ecd01a84051350279815ead1b40b04e306bc1054692: Status 404 returned error can't find the container with id 9c2e6b21246927ca4a9f3ecd01a84051350279815ead1b40b04e306bc1054692
Nov 02 12:39:55.924688 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:55.924603    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7" event={"ID":"8c8f58de-36cb-403f-ba87-965c05827381","Type":"ContainerStarted","Data":"7c177ea8148876cf1f7522b8c92fc859eea6b7f080cc62f071e39c91baba9b07"}
Nov 02 12:39:55.924688 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:55.924663    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7" event={"ID":"8c8f58de-36cb-403f-ba87-965c05827381","Type":"ContainerStarted","Data":"9c2e6b21246927ca4a9f3ecd01a84051350279815ead1b40b04e306bc1054692"}
Nov 02 12:39:55.925107 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:55.924691    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:39:56.047007 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:39:56.046945    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7" podStartSLOduration=2.046932106 podStartE2EDuration="2.046932106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:39:56.045272684 +0000 UTC m=+2624.608981094" watchObservedRunningTime="2026-11-02 12:39:56.046932106 +0000 UTC m=+2624.610640514"
Nov 02 12:39:58.294422 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:39:58.294388    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:40:06.932891 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:06.932862    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:40:15.304059 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:15.304022    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:40:28.291059 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:40:28.290962    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:40:30.832912 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:30.832870    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"]
Nov 02 12:40:30.838564 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:30.838533    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"
Nov 02 12:40:30.894518 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:30.894489    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"]
Nov 02 12:40:30.919502 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:30.919464    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk6jr\" (UniqueName: \"kubernetes.io/projected/b2c37bdd-7327-4b35-900c-7aa9065c6466-kube-api-access-fk6jr\") pod \"maas-api-revoke-sub-87ba9b34-lxqhd\" (UID: \"b2c37bdd-7327-4b35-900c-7aa9065c6466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"
Nov 02 12:40:30.919502 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:30.919517    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c37bdd-7327-4b35-900c-7aa9065c6466-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87ba9b34-lxqhd\" (UID: \"b2c37bdd-7327-4b35-900c-7aa9065c6466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"
Nov 02 12:40:30.919747 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:30.919646    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c37bdd-7327-4b35-900c-7aa9065c6466-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87ba9b34-lxqhd\" (UID: \"b2c37bdd-7327-4b35-900c-7aa9065c6466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"
Nov 02 12:40:30.942165 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:30.942142    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:40:30.942304 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:30.942161    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:40:30.942304 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:30.942163    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:40:30.942649 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:30.942626    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:40:31.020897 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:31.020874    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk6jr\" (UniqueName: \"kubernetes.io/projected/b2c37bdd-7327-4b35-900c-7aa9065c6466-kube-api-access-fk6jr\") pod \"maas-api-revoke-sub-87ba9b34-lxqhd\" (UID: \"b2c37bdd-7327-4b35-900c-7aa9065c6466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"
Nov 02 12:40:31.021094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:31.020915    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c37bdd-7327-4b35-900c-7aa9065c6466-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87ba9b34-lxqhd\" (UID: \"b2c37bdd-7327-4b35-900c-7aa9065c6466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"
Nov 02 12:40:31.021094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:31.020988    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c37bdd-7327-4b35-900c-7aa9065c6466-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87ba9b34-lxqhd\" (UID: \"b2c37bdd-7327-4b35-900c-7aa9065c6466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"
Nov 02 12:40:31.021565 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:31.021546    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c37bdd-7327-4b35-900c-7aa9065c6466-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87ba9b34-lxqhd\" (UID: \"b2c37bdd-7327-4b35-900c-7aa9065c6466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"
Nov 02 12:40:31.023334 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:31.023313    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c37bdd-7327-4b35-900c-7aa9065c6466-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87ba9b34-lxqhd\" (UID: \"b2c37bdd-7327-4b35-900c-7aa9065c6466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"
Nov 02 12:40:31.082858 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:31.082828    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk6jr\" (UniqueName: \"kubernetes.io/projected/b2c37bdd-7327-4b35-900c-7aa9065c6466-kube-api-access-fk6jr\") pod \"maas-api-revoke-sub-87ba9b34-lxqhd\" (UID: \"b2c37bdd-7327-4b35-900c-7aa9065c6466\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"
Nov 02 12:40:31.153923 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:31.153857    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"
Nov 02 12:40:31.528763 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:31.528739    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"]
Nov 02 12:40:31.530514 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:40:31.530485    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c37bdd_7327_4b35_900c_7aa9065c6466.slice/crio-d57da115070314ae00d6169b97691e1a408338b035b6453413273fb4fb8415dc WatchSource:0}: Error finding container d57da115070314ae00d6169b97691e1a408338b035b6453413273fb4fb8415dc: Status 404 returned error can't find the container with id d57da115070314ae00d6169b97691e1a408338b035b6453413273fb4fb8415dc
Nov 02 12:40:32.106443 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:32.106409    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd" event={"ID":"b2c37bdd-7327-4b35-900c-7aa9065c6466","Type":"ContainerStarted","Data":"2a8c90006099a0f405b86bc1f178c2efb1ef70da8be6145eb20fdc1add84f800"}
Nov 02 12:40:32.106805 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:32.106466    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd" event={"ID":"b2c37bdd-7327-4b35-900c-7aa9065c6466","Type":"ContainerStarted","Data":"d57da115070314ae00d6169b97691e1a408338b035b6453413273fb4fb8415dc"}
Nov 02 12:40:32.224745 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:32.224697    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd" podStartSLOduration=2.224683719 podStartE2EDuration="2.224683719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:32.223309968 +0000 UTC m=+2660.787018404" watchObservedRunningTime="2026-11-02 12:40:32.224683719 +0000 UTC m=+2660.788392128"
Nov 02 12:40:33.113481 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:33.113451    4464 generic.go:356] "Generic (PLEG): container finished" podID="b2c37bdd-7327-4b35-900c-7aa9065c6466" containerID="2a8c90006099a0f405b86bc1f178c2efb1ef70da8be6145eb20fdc1add84f800" exitCode=0
Nov 02 12:40:33.113895 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:33.113495    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd" event={"ID":"b2c37bdd-7327-4b35-900c-7aa9065c6466","Type":"ContainerDied","Data":"2a8c90006099a0f405b86bc1f178c2efb1ef70da8be6145eb20fdc1add84f800"}
Nov 02 12:40:34.250182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:34.250156    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"
Nov 02 12:40:34.351563 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:34.351535    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2c37bdd-7327-4b35-900c-7aa9065c6466-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c37bdd-7327-4b35-900c-7aa9065c6466-maas-api-xks-ca\") pod \"b2c37bdd-7327-4b35-900c-7aa9065c6466\" (UID: \"b2c37bdd-7327-4b35-900c-7aa9065c6466\") "
Nov 02 12:40:34.351711 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:34.351579    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c37bdd-7327-4b35-900c-7aa9065c6466-kube-api-access-fk6jr\" (UniqueName: \"kubernetes.io/projected/b2c37bdd-7327-4b35-900c-7aa9065c6466-kube-api-access-fk6jr\") pod \"b2c37bdd-7327-4b35-900c-7aa9065c6466\" (UID: \"b2c37bdd-7327-4b35-900c-7aa9065c6466\") "
Nov 02 12:40:34.351711 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:34.351656    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c37bdd-7327-4b35-900c-7aa9065c6466-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c37bdd-7327-4b35-900c-7aa9065c6466-maas-api-openshift-ca\") pod \"b2c37bdd-7327-4b35-900c-7aa9065c6466\" (UID: \"b2c37bdd-7327-4b35-900c-7aa9065c6466\") "
Nov 02 12:40:34.352086 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:34.352066    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c37bdd-7327-4b35-900c-7aa9065c6466-maas-api-openshift-ca" pod "b2c37bdd-7327-4b35-900c-7aa9065c6466" (UID: "b2c37bdd-7327-4b35-900c-7aa9065c6466"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:40:34.353568 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:34.353544    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c37bdd-7327-4b35-900c-7aa9065c6466-maas-api-xks-ca" pod "b2c37bdd-7327-4b35-900c-7aa9065c6466" (UID: "b2c37bdd-7327-4b35-900c-7aa9065c6466"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:40:34.353677 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:34.353590    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c37bdd-7327-4b35-900c-7aa9065c6466-kube-api-access-fk6jr" pod "b2c37bdd-7327-4b35-900c-7aa9065c6466" (UID: "b2c37bdd-7327-4b35-900c-7aa9065c6466"). InnerVolumeSpecName "kube-api-access-fk6jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:40:34.453253 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:34.453179    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c37bdd-7327-4b35-900c-7aa9065c6466-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:40:34.453253 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:34.453216    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk6jr\" (UniqueName: \"kubernetes.io/projected/b2c37bdd-7327-4b35-900c-7aa9065c6466-kube-api-access-fk6jr\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:40:34.453253 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:34.453227    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c37bdd-7327-4b35-900c-7aa9065c6466-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:40:35.124631 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:35.124602    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"
Nov 02 12:40:35.124631 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:35.124617    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd" event={"ID":"b2c37bdd-7327-4b35-900c-7aa9065c6466","Type":"ContainerDied","Data":"d57da115070314ae00d6169b97691e1a408338b035b6453413273fb4fb8415dc"}
Nov 02 12:40:35.124819 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:35.124659    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d57da115070314ae00d6169b97691e1a408338b035b6453413273fb4fb8415dc"
Nov 02 12:40:43.047724 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:43.047691    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"]
Nov 02 12:40:47.612146 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:47.612112    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:40:48.048278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:48.048200    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7" podUID="8c8f58de-36cb-403f-ba87-965c05827381" containerName="payload-processing" containerID="cri-o://7c177ea8148876cf1f7522b8c92fc859eea6b7f080cc62f071e39c91baba9b07" gracePeriod=25
Nov 02 12:40:48.195957 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:48.195928    4464 generic.go:356] "Generic (PLEG): container finished" podID="8c8f58de-36cb-403f-ba87-965c05827381" containerID="7c177ea8148876cf1f7522b8c92fc859eea6b7f080cc62f071e39c91baba9b07" exitCode=0
Nov 02 12:40:48.196132 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:48.195992    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7" event={"ID":"8c8f58de-36cb-403f-ba87-965c05827381","Type":"ContainerDied","Data":"7c177ea8148876cf1f7522b8c92fc859eea6b7f080cc62f071e39c91baba9b07"}
Nov 02 12:40:48.299357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:48.299301    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:40:48.386958 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:48.384655    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8f58de-36cb-403f-ba87-965c05827381-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c8f58de-36cb-403f-ba87-965c05827381-provider-credentials\") pod \"8c8f58de-36cb-403f-ba87-965c05827381\" (UID: \"8c8f58de-36cb-403f-ba87-965c05827381\") "
Nov 02 12:40:48.386958 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:48.384722    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8f58de-36cb-403f-ba87-965c05827381-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8f58de-36cb-403f-ba87-965c05827381-plugins-config-volume\") pod \"8c8f58de-36cb-403f-ba87-965c05827381\" (UID: \"8c8f58de-36cb-403f-ba87-965c05827381\") "
Nov 02 12:40:48.386958 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:48.384773    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8f58de-36cb-403f-ba87-965c05827381-kube-api-access-hhrgd\" (UniqueName: \"kubernetes.io/projected/8c8f58de-36cb-403f-ba87-965c05827381-kube-api-access-hhrgd\") pod \"8c8f58de-36cb-403f-ba87-965c05827381\" (UID: \"8c8f58de-36cb-403f-ba87-965c05827381\") "
Nov 02 12:40:48.386958 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:48.384834    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8f58de-36cb-403f-ba87-965c05827381-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c8f58de-36cb-403f-ba87-965c05827381-routing-overlay-volume\") pod \"8c8f58de-36cb-403f-ba87-965c05827381\" (UID: \"8c8f58de-36cb-403f-ba87-965c05827381\") "
Nov 02 12:40:48.386958 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:48.385726    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8f58de-36cb-403f-ba87-965c05827381-routing-overlay-volume" pod "8c8f58de-36cb-403f-ba87-965c05827381" (UID: "8c8f58de-36cb-403f-ba87-965c05827381"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:40:48.389560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:48.388530    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8f58de-36cb-403f-ba87-965c05827381-provider-credentials" pod "8c8f58de-36cb-403f-ba87-965c05827381" (UID: "8c8f58de-36cb-403f-ba87-965c05827381"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:40:48.389560 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:48.388901    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8f58de-36cb-403f-ba87-965c05827381-plugins-config-volume" pod "8c8f58de-36cb-403f-ba87-965c05827381" (UID: "8c8f58de-36cb-403f-ba87-965c05827381"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:40:48.392938 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:48.392914    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8f58de-36cb-403f-ba87-965c05827381-kube-api-access-hhrgd" pod "8c8f58de-36cb-403f-ba87-965c05827381" (UID: "8c8f58de-36cb-403f-ba87-965c05827381"). InnerVolumeSpecName "kube-api-access-hhrgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:40:48.486660 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:48.486620    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c8f58de-36cb-403f-ba87-965c05827381-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:40:48.486660 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:48.486658    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c8f58de-36cb-403f-ba87-965c05827381-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:40:48.486868 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:48.486668    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhrgd\" (UniqueName: \"kubernetes.io/projected/8c8f58de-36cb-403f-ba87-965c05827381-kube-api-access-hhrgd\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:40:48.486868 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:48.486677    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c8f58de-36cb-403f-ba87-965c05827381-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:40:49.203565 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:49.203529    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7" event={"ID":"8c8f58de-36cb-403f-ba87-965c05827381","Type":"ContainerDied","Data":"9c2e6b21246927ca4a9f3ecd01a84051350279815ead1b40b04e306bc1054692"}
Nov 02 12:40:49.204038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:49.203582    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"
Nov 02 12:40:49.204038 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:49.203617    4464 scope.go:122] "RemoveContainer" containerID="7c177ea8148876cf1f7522b8c92fc859eea6b7f080cc62f071e39c91baba9b07"
Nov 02 12:40:49.399953 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:49.399921    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"]
Nov 02 12:40:49.405669 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:49.405644    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7cbc96pb7"]
Nov 02 12:40:49.581862 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:49.581831    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8f58de-36cb-403f-ba87-965c05827381" path="/var/lib/kubelet/pods/8c8f58de-36cb-403f-ba87-965c05827381/volumes"
Nov 02 12:40:50.515396 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.515361    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"]
Nov 02 12:40:50.516043 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.516023    4464 cpu_manager.go:414] "containerMap: removing container" podUID="8c8f58de-36cb-403f-ba87-965c05827381" containerName="payload-processing"
Nov 02 12:40:50.516152 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.516043    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8f58de-36cb-403f-ba87-965c05827381" containerName="payload-processing"
Nov 02 12:40:50.516152 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.516111    4464 cpu_manager.go:414] "containerMap: removing container" podUID="b2c37bdd-7327-4b35-900c-7aa9065c6466" containerName="revoke-keys"
Nov 02 12:40:50.516152 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.516120    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c37bdd-7327-4b35-900c-7aa9065c6466" containerName="revoke-keys"
Nov 02 12:40:50.516315 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.516253    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c37bdd-7327-4b35-900c-7aa9065c6466" containerName="revoke-keys"
Nov 02 12:40:50.516315 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.516272    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8f58de-36cb-403f-ba87-965c05827381" containerName="payload-processing"
Nov 02 12:40:50.520809 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.520789    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:40:50.575064 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.574962    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"kube-root-ca.crt\""
Nov 02 12:40:50.605163 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.605131    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr5cq\" (UniqueName: \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-kube-api-access-kr5cq\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:40:50.605297 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.605176    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:40:50.605366 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.605347    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/242becf2-522b-492b-b390-e56675435069-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:40:50.605405 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.605377    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/242becf2-522b-492b-b390-e56675435069-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:40:50.624893 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.624867    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"openshift-service-ca.crt\""
Nov 02 12:40:50.625057 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.624993    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"agc-cred-1352e9f85352\""
Nov 02 12:40:50.625057 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.624996    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6ac03f\""
Nov 02 12:40:50.625192 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.625096    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"routing-overlay\""
Nov 02 12:40:50.625192 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.625145    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"payload-processing-external-model-e2e--75a83f02-dockercfg-hlj5d\""
Nov 02 12:40:50.627466 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.627449    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"]
Nov 02 12:40:50.706082 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.706058    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/242becf2-522b-492b-b390-e56675435069-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:40:50.706206 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.706087    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/242becf2-522b-492b-b390-e56675435069-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:40:50.706206 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.706109    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr5cq\" (UniqueName: \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-kube-api-access-kr5cq\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:40:50.706206 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.706130    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:40:50.706343 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:40:50.706229    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-1352e9f85352: secret "agc-cred-1352e9f85352" not found
Nov 02 12:40:50.706343 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:40:50.706239    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv: secret "agc-cred-1352e9f85352" not found
Nov 02 12:40:50.706343 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:40:50.706291    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials podName:242becf2-522b-492b-b390-e56675435069 nodeName:}" failed. No retries permitted until 2026-11-02 12:40:51.206276721 +0000 UTC m=+2679.769985122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv" (UID: "242becf2-522b-492b-b390-e56675435069") : secret "agc-cred-1352e9f85352" not found
Nov 02 12:40:50.706882 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.706861    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/242becf2-522b-492b-b390-e56675435069-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:40:50.706956 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.706943    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/242becf2-522b-492b-b390-e56675435069-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:40:50.768578 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:50.768513    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr5cq\" (UniqueName: \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-kube-api-access-kr5cq\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:40:51.210425 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:51.210343    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:40:51.210589 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:40:51.210482    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-1352e9f85352: secret "agc-cred-1352e9f85352" not found
Nov 02 12:40:51.210589 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:40:51.210502    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv: secret "agc-cred-1352e9f85352" not found
Nov 02 12:40:51.210589 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:40:51.210551    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials podName:242becf2-522b-492b-b390-e56675435069 nodeName:}" failed. No retries permitted until 2026-11-02 12:40:52.210538354 +0000 UTC m=+2680.774246757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv" (UID: "242becf2-522b-492b-b390-e56675435069") : secret "agc-cred-1352e9f85352" not found
Nov 02 12:40:51.807404 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:51.807368    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"]
Nov 02 12:40:51.811378 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:51.811356    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"
Nov 02 12:40:51.866587 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:51.866554    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"e2e-cfl-970074-provider-cred\""
Nov 02 12:40:51.869512 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:51.869485    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"]
Nov 02 12:40:51.917388 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:51.917358    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g\" (UID: \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"
Nov 02 12:40:51.917522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:51.917421    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g\" (UID: \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"
Nov 02 12:40:51.917522 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:51.917492    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g\" (UID: \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"
Nov 02 12:40:51.917664 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:51.917560    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-675cc\" (UniqueName: \"kubernetes.io/projected/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-kube-api-access-675cc\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g\" (UID: \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"
Nov 02 12:40:52.019071 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:52.019041    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g\" (UID: \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"
Nov 02 12:40:52.019212 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:52.019113    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g\" (UID: \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"
Nov 02 12:40:52.019212 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:52.019143    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g\" (UID: \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"
Nov 02 12:40:52.019212 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:52.019193    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-675cc\" (UniqueName: \"kubernetes.io/projected/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-kube-api-access-675cc\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g\" (UID: \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"
Nov 02 12:40:52.019769 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:52.019743    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g\" (UID: \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"
Nov 02 12:40:52.019885 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:52.019860    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g\" (UID: \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"
Nov 02 12:40:52.021435 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:52.021411    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g\" (UID: \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"
Nov 02 12:40:52.083049 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:52.082993    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-675cc\" (UniqueName: \"kubernetes.io/projected/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-kube-api-access-675cc\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g\" (UID: \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"
Nov 02 12:40:52.124126 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:52.124101    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"
Nov 02 12:40:52.220887 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:52.220859    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:40:52.221067 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:40:52.221044    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-1352e9f85352: secret "agc-cred-1352e9f85352" not found
Nov 02 12:40:52.221115 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:40:52.221071    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv: secret "agc-cred-1352e9f85352" not found
Nov 02 12:40:52.221153 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:40:52.221141    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials podName:242becf2-522b-492b-b390-e56675435069 nodeName:}" failed. No retries permitted until 2026-11-02 12:40:54.221122108 +0000 UTC m=+2682.784830511 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv" (UID: "242becf2-522b-492b-b390-e56675435069") : secret "agc-cred-1352e9f85352" not found
Nov 02 12:40:52.496553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:52.496525    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"]
Nov 02 12:40:52.497448 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:40:52.497424    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f830a9_2aa4_4720_a5a8_3ceb98d69c3a.slice/crio-d91f3bf274c441a3f0b63579bce23486913e714a480c83ff3a8fd81e4f8dcd7c WatchSource:0}: Error finding container d91f3bf274c441a3f0b63579bce23486913e714a480c83ff3a8fd81e4f8dcd7c: Status 404 returned error can't find the container with id d91f3bf274c441a3f0b63579bce23486913e714a480c83ff3a8fd81e4f8dcd7c
Nov 02 12:40:53.228035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:53.228001    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" event={"ID":"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a","Type":"ContainerStarted","Data":"f423f7b0c5060cebe5107bc8db224898181062300f8c706772668f715d053581"}
Nov 02 12:40:53.228448 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:53.228051    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" event={"ID":"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a","Type":"ContainerStarted","Data":"d91f3bf274c441a3f0b63579bce23486913e714a480c83ff3a8fd81e4f8dcd7c"}
Nov 02 12:40:53.228448 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:53.228205    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"
Nov 02 12:40:53.347702 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:53.347651    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podStartSLOduration=2.347634771 podStartE2EDuration="2.347634771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:53.346428992 +0000 UTC m=+2681.910137402" watchObservedRunningTime="2026-11-02 12:40:53.347634771 +0000 UTC m=+2681.911343178"
Nov 02 12:40:54.243089 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:54.243055    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:40:54.243489 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:40:54.243187    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-1352e9f85352: secret "agc-cred-1352e9f85352" not found
Nov 02 12:40:54.243489 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:40:54.243202    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv: secret "agc-cred-1352e9f85352" not found
Nov 02 12:40:54.243489 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:40:54.243253    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials podName:242becf2-522b-492b-b390-e56675435069 nodeName:}" failed. No retries permitted until 2026-11-02 12:40:58.24323698 +0000 UTC m=+2686.806945383 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv" (UID: "242becf2-522b-492b-b390-e56675435069") : secret "agc-cred-1352e9f85352" not found
Nov 02 12:40:58.280711 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:40:58.280668    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:40:58.281223 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:40:58.280819    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-1352e9f85352: secret "agc-cred-1352e9f85352" not found
Nov 02 12:40:58.281223 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:40:58.280840    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv: secret "agc-cred-1352e9f85352" not found
Nov 02 12:40:58.281223 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:40:58.280898    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials podName:242becf2-522b-492b-b390-e56675435069 nodeName:}" failed. No retries permitted until 2026-11-02 12:41:06.280881428 +0000 UTC m=+2694.844589815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv" (UID: "242becf2-522b-492b-b390-e56675435069") : secret "agc-cred-1352e9f85352" not found
Nov 02 12:40:58.292497 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:40:58.292465    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:41:00.790224 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:41:00.790186    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:41:04.235776 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:41:04.235738    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:41:06.347470 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:41:06.347438    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:41:06.347854 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:41:06.347559    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-1352e9f85352: secret "agc-cred-1352e9f85352" not found
Nov 02 12:41:06.347854 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:41:06.347570    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv: secret "agc-cred-1352e9f85352" not found
Nov 02 12:41:06.347854 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:41:06.347617    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials podName:242becf2-522b-492b-b390-e56675435069 nodeName:}" failed. No retries permitted until 2026-11-02 12:41:22.347603486 +0000 UTC m=+2710.911311873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv" (UID: "242becf2-522b-492b-b390-e56675435069") : secret "agc-cred-1352e9f85352" not found
Nov 02 12:41:11.952881 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:41:11.952852    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:41:11.969813 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:41:11.969789    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:41:11.990696 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:41:11.990676    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:41:12.007147 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:41:12.007129    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:41:12.905859 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:41:12.905823    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:41:14.235186 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:41:14.235156    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:41:22.398811 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:41:22.398775    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:41:22.399343 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:41:22.398944    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-1352e9f85352: secret "agc-cred-1352e9f85352" not found
Nov 02 12:41:22.399343 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:41:22.398967    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv: secret "agc-cred-1352e9f85352" not found
Nov 02 12:41:22.399343 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:41:22.399088    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials podName:242becf2-522b-492b-b390-e56675435069 nodeName:}" failed. No retries permitted until 2026-11-02 12:41:54.399067764 +0000 UTC m=+2742.962776167 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv" (UID: "242becf2-522b-492b-b390-e56675435069") : secret "agc-cred-1352e9f85352" not found
Nov 02 12:41:24.235290 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:41:24.235252    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:41:28.292283 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:41:28.292246    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:41:34.235321 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:41:34.235275    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:41:44.235286 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:41:44.235241    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:41:54.234864 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:41:54.234827    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:41:54.420920 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:41:54.420884    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:41:54.421122 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:41:54.421024    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-1352e9f85352: secret "agc-cred-1352e9f85352" not found
Nov 02 12:41:54.421122 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:41:54.421037    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv: secret "agc-cred-1352e9f85352" not found
Nov 02 12:41:54.421122 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:41:54.421086    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials podName:242becf2-522b-492b-b390-e56675435069 nodeName:}" failed. No retries permitted until 2026-11-02 12:42:58.421071431 +0000 UTC m=+2806.984779817 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv" (UID: "242becf2-522b-492b-b390-e56675435069") : secret "agc-cred-1352e9f85352" not found
Nov 02 12:41:58.291511 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:41:58.291477    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:42:04.235298 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:42:04.235257    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:42:14.234421 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:42:14.234379    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:42:20.576283 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:42:20.576244    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:42:28.291271 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:42:28.291227    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:42:30.577195 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:42:30.577155    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:42:40.576502 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:42:40.576459    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:42:50.576338 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:42:50.576305    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:42:53.533805 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:42:53.533767    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv" podUID="242becf2-522b-492b-b390-e56675435069"
Nov 02 12:42:53.850383 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:42:53.850307    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:42:58.292058 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:42:58.292018    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:42:58.458341 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:42:58.458303    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:42:58.458545 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:42:58.458434    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-1352e9f85352: secret "agc-cred-1352e9f85352" not found
Nov 02 12:42:58.458545 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:42:58.458446    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv: secret "agc-cred-1352e9f85352" not found
Nov 02 12:42:58.458545 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:42:58.458501    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials podName:242becf2-522b-492b-b390-e56675435069 nodeName:}" failed. No retries permitted until 2026-11-02 12:45:00.458485542 +0000 UTC m=+2929.022193944 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv" (UID: "242becf2-522b-492b-b390-e56675435069") : secret "agc-cred-1352e9f85352" not found
Nov 02 12:43:00.576965 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:43:00.576923    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:43:10.576287 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:43:10.576241    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:43:20.576802 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:43:20.576760    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:43:22.048315 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:43:22.048268    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"]
Nov 02 12:43:22.098846 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:43:22.098816    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77cea573-kk5ln"]
Nov 02 12:43:23.581369 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:43:23.581332    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b09daaa3-1e6f-4bb0-9437-6d440e085446" path="/var/lib/kubelet/pods/b09daaa3-1e6f-4bb0-9437-6d440e085446/volumes"
Nov 02 12:43:28.291792 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:43:28.291714    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:43:30.576393 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:43:30.576359    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:43:35.059086 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:43:35.059048    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"]
Nov 02 12:43:35.109324 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:43:35.109291    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cde896e-jgd98"]
Nov 02 12:43:35.581355 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:43:35.581318    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1514196f-2354-4e14-a8b6-939cd6f1346b" path="/var/lib/kubelet/pods/1514196f-2354-4e14-a8b6-939cd6f1346b/volumes"
Nov 02 12:43:38.576197 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:43:38.576157    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:43:47.047915 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:43:47.047878    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"]
Nov 02 12:43:47.098616 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:43:47.098589    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42b888e2-whh7m"]
Nov 02 12:43:47.580471 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:43:47.580441    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07429504-309b-40e8-b1d5-9c7bde3b683d" path="/var/lib/kubelet/pods/07429504-309b-40e8-b1d5-9c7bde3b683d/volumes"
Nov 02 12:43:48.576382 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:43:48.576345    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:43:58.291452 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:43:58.291409    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:43:58.576780 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:43:58.576694    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:44:02.058913 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:44:02.058870    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"]
Nov 02 12:44:02.109664 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:44:02.109631    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6472f25-mkqn6"]
Nov 02 12:44:03.581406 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:44:03.581374    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a660cf-6e8c-4bca-aadb-5c99ff72c6b2" path="/var/lib/kubelet/pods/89a660cf-6e8c-4bca-aadb-5c99ff72c6b2/volumes"
Nov 02 12:44:08.577121 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:44:08.577081    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:44:15.048306 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:44:15.048262    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"]
Nov 02 12:44:15.098828 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:44:15.098791    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5207ddd4-ctdb7"]
Nov 02 12:44:15.581199 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:44:15.581158    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4fc0803-0ae8-4cf3-8b47-b6b1f838b089" path="/var/lib/kubelet/pods/f4fc0803-0ae8-4cf3-8b47-b6b1f838b089/volumes"
Nov 02 12:44:18.577021 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:44:18.576968    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:44:28.048803 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:44:28.048766    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"]
Nov 02 12:44:28.098809 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:44:28.098772    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d757018-gstjm"]
Nov 02 12:44:28.292739 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:44:28.292698    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:44:28.576268 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:44:28.576237    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:44:29.581097 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:44:29.581062    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00497b1b-8136-453b-a1a9-2b136c39ba67" path="/var/lib/kubelet/pods/00497b1b-8136-453b-a1a9-2b136c39ba67/volumes"
Nov 02 12:44:38.577095 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:44:38.577057    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:44:43.576067 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:44:43.576025    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:44:53.576204 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:44:53.576167    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:44:56.851311 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:44:56.851219    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv" podUID="242becf2-522b-492b-b390-e56675435069"
Nov 02 12:44:57.512502 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:44:57.512447    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:44:58.294941 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:44:58.294901    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:45:00.187307 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.187275    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv"]
Nov 02 12:45:00.191247 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.191228    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv"]
Nov 02 12:45:00.191352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.191339    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv"
Nov 02 12:45:00.194802 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.194786    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv"
Nov 02 12:45:00.245015 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.244986    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:45:00.245015 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.244992    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:45:00.245224 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.245046    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:45:00.275163 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.275125    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sffjt\" (UniqueName: \"kubernetes.io/projected/04f90f67-c34e-4f47-8d8f-953837c63533-kube-api-access-sffjt\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv\" (UID: \"04f90f67-c34e-4f47-8d8f-953837c63533\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv"
Nov 02 12:45:00.275311 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.275179    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6vd2\" (UniqueName: \"kubernetes.io/projected/f94057a7-f3f0-4b15-99aa-475d330452a1-kube-api-access-h6vd2\") pod \"maas-api-key-cleanup-29893725-2vvzv\" (UID: \"f94057a7-f3f0-4b15-99aa-475d330452a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv"
Nov 02 12:45:00.275311 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.275298    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f90f67-c34e-4f47-8d8f-953837c63533-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv\" (UID: \"04f90f67-c34e-4f47-8d8f-953837c63533\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv"
Nov 02 12:45:00.275391 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.275359    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94057a7-f3f0-4b15-99aa-475d330452a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893725-2vvzv\" (UID: \"f94057a7-f3f0-4b15-99aa-475d330452a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv"
Nov 02 12:45:00.297943 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.297908    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv"]
Nov 02 12:45:00.299807 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.299781    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv"]
Nov 02 12:45:00.376133 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.376100    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94057a7-f3f0-4b15-99aa-475d330452a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893725-2vvzv\" (UID: \"f94057a7-f3f0-4b15-99aa-475d330452a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv"
Nov 02 12:45:00.376357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.376171    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sffjt\" (UniqueName: \"kubernetes.io/projected/04f90f67-c34e-4f47-8d8f-953837c63533-kube-api-access-sffjt\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv\" (UID: \"04f90f67-c34e-4f47-8d8f-953837c63533\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv"
Nov 02 12:45:00.376357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.376208    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6vd2\" (UniqueName: \"kubernetes.io/projected/f94057a7-f3f0-4b15-99aa-475d330452a1-kube-api-access-h6vd2\") pod \"maas-api-key-cleanup-29893725-2vvzv\" (UID: \"f94057a7-f3f0-4b15-99aa-475d330452a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv"
Nov 02 12:45:00.376357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.376269    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f90f67-c34e-4f47-8d8f-953837c63533-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv\" (UID: \"04f90f67-c34e-4f47-8d8f-953837c63533\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv"
Nov 02 12:45:00.376830 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.376804    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94057a7-f3f0-4b15-99aa-475d330452a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893725-2vvzv\" (UID: \"f94057a7-f3f0-4b15-99aa-475d330452a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv"
Nov 02 12:45:00.376923 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.376862    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f90f67-c34e-4f47-8d8f-953837c63533-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv\" (UID: \"04f90f67-c34e-4f47-8d8f-953837c63533\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv"
Nov 02 12:45:00.438645 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.438575    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6vd2\" (UniqueName: \"kubernetes.io/projected/f94057a7-f3f0-4b15-99aa-475d330452a1-kube-api-access-h6vd2\") pod \"maas-api-key-cleanup-29893725-2vvzv\" (UID: \"f94057a7-f3f0-4b15-99aa-475d330452a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv"
Nov 02 12:45:00.438783 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.438769    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sffjt\" (UniqueName: \"kubernetes.io/projected/04f90f67-c34e-4f47-8d8f-953837c63533-kube-api-access-sffjt\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv\" (UID: \"04f90f67-c34e-4f47-8d8f-953837c63533\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv"
Nov 02 12:45:00.477790 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.477760    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv\" (UID: \"242becf2-522b-492b-b390-e56675435069\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:45:00.477951 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:45:00.477902    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-1352e9f85352: secret "agc-cred-1352e9f85352" not found
Nov 02 12:45:00.477951 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:45:00.477920    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv: secret "agc-cred-1352e9f85352" not found
Nov 02 12:45:00.478048 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:45:00.477991    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials podName:242becf2-522b-492b-b390-e56675435069 nodeName:}" failed. No retries permitted until 2026-11-02 12:47:02.477957483 +0000 UTC m=+3051.041665870 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv" (UID: "242becf2-522b-492b-b390-e56675435069") : secret "agc-cred-1352e9f85352" not found
Nov 02 12:45:00.505609 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.505581    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv"
Nov 02 12:45:00.511756 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.511733    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv"
Nov 02 12:45:00.884646 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.884616    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv"]
Nov 02 12:45:00.886445 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:45:00.886422    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f90f67_c34e_4f47_8d8f_953837c63533.slice/crio-3d3ce42729d301f856420c78c1c6b8ada94056bf9c0d4986051fc87737f560a7 WatchSource:0}: Error finding container 3d3ce42729d301f856420c78c1c6b8ada94056bf9c0d4986051fc87737f560a7: Status 404 returned error can't find the container with id 3d3ce42729d301f856420c78c1c6b8ada94056bf9c0d4986051fc87737f560a7
Nov 02 12:45:00.935251 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:00.935225    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv"]
Nov 02 12:45:00.936596 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:45:00.936570    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94057a7_f3f0_4b15_99aa_475d330452a1.slice/crio-311eaec87c95a3040ab8b98dacfce9b814084f761e672225f2ff70b6e1866fa9 WatchSource:0}: Error finding container 311eaec87c95a3040ab8b98dacfce9b814084f761e672225f2ff70b6e1866fa9: Status 404 returned error can't find the container with id 311eaec87c95a3040ab8b98dacfce9b814084f761e672225f2ff70b6e1866fa9
Nov 02 12:45:01.534428 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:01.534390    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv" event={"ID":"04f90f67-c34e-4f47-8d8f-953837c63533","Type":"ContainerStarted","Data":"b056c3077eeb8ece89185770a4843053946e46271fbab43a8ec6ac3498ff8089"}
Nov 02 12:45:01.534861 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:01.534448    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv" event={"ID":"04f90f67-c34e-4f47-8d8f-953837c63533","Type":"ContainerStarted","Data":"3d3ce42729d301f856420c78c1c6b8ada94056bf9c0d4986051fc87737f560a7"}
Nov 02 12:45:01.536032 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:01.536005    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv" event={"ID":"f94057a7-f3f0-4b15-99aa-475d330452a1","Type":"ContainerStarted","Data":"d79506581d2fcb6cb678bf8c769acc5bd35f92e6a5c1329e76ec9dbc8ff5cdab"}
Nov 02 12:45:01.536152 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:01.536047    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv" event={"ID":"f94057a7-f3f0-4b15-99aa-475d330452a1","Type":"ContainerStarted","Data":"311eaec87c95a3040ab8b98dacfce9b814084f761e672225f2ff70b6e1866fa9"}
Nov 02 12:45:01.652244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:01.652193    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv" podStartSLOduration=1.65218061 podStartE2EDuration="1.65218061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:01.651445493 +0000 UTC m=+2930.215153903" watchObservedRunningTime="2026-11-02 12:45:01.65218061 +0000 UTC m=+2930.215889018"
Nov 02 12:45:01.770097 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:01.770047    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv" podStartSLOduration=1.770032576 podStartE2EDuration="1.770032576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:01.767732766 +0000 UTC m=+2930.331441175" watchObservedRunningTime="2026-11-02 12:45:01.770032576 +0000 UTC m=+2930.333740985"
Nov 02 12:45:02.542804 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:02.542770    4464 generic.go:356] "Generic (PLEG): container finished" podID="f94057a7-f3f0-4b15-99aa-475d330452a1" containerID="d79506581d2fcb6cb678bf8c769acc5bd35f92e6a5c1329e76ec9dbc8ff5cdab" exitCode=60
Nov 02 12:45:02.543309 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:02.542856    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv" event={"ID":"f94057a7-f3f0-4b15-99aa-475d330452a1","Type":"ContainerDied","Data":"d79506581d2fcb6cb678bf8c769acc5bd35f92e6a5c1329e76ec9dbc8ff5cdab"}
Nov 02 12:45:02.543309 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:02.543134    4464 scope.go:122] "RemoveContainer" containerID="d79506581d2fcb6cb678bf8c769acc5bd35f92e6a5c1329e76ec9dbc8ff5cdab"
Nov 02 12:45:02.544543 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:02.544428    4464 generic.go:356] "Generic (PLEG): container finished" podID="04f90f67-c34e-4f47-8d8f-953837c63533" containerID="b056c3077eeb8ece89185770a4843053946e46271fbab43a8ec6ac3498ff8089" exitCode=60
Nov 02 12:45:02.544543 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:02.544468    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv" event={"ID":"04f90f67-c34e-4f47-8d8f-953837c63533","Type":"ContainerDied","Data":"b056c3077eeb8ece89185770a4843053946e46271fbab43a8ec6ac3498ff8089"}
Nov 02 12:45:02.544659 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:02.544646    4464 scope.go:122] "RemoveContainer" containerID="b056c3077eeb8ece89185770a4843053946e46271fbab43a8ec6ac3498ff8089"
Nov 02 12:45:03.551400 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:03.551364    4464 generic.go:356] "Generic (PLEG): container finished" podID="f94057a7-f3f0-4b15-99aa-475d330452a1" containerID="4dce19c6d13c6476560e5aad6f8c6048fae0b72ad983778c9b5001b6465841d7" exitCode=60
Nov 02 12:45:03.551863 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:03.551447    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv" event={"ID":"f94057a7-f3f0-4b15-99aa-475d330452a1","Type":"ContainerDied","Data":"4dce19c6d13c6476560e5aad6f8c6048fae0b72ad983778c9b5001b6465841d7"}
Nov 02 12:45:03.551863 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:03.551534    4464 scope.go:122] "RemoveContainer" containerID="d79506581d2fcb6cb678bf8c769acc5bd35f92e6a5c1329e76ec9dbc8ff5cdab"
Nov 02 12:45:03.551863 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:03.551783    4464 scope.go:122] "RemoveContainer" containerID="4dce19c6d13c6476560e5aad6f8c6048fae0b72ad983778c9b5001b6465841d7"
Nov 02 12:45:03.552088 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:45:03.552066    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893725-2vvzv_odh-ai-gateway-infra(f94057a7-f3f0-4b15-99aa-475d330452a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv" podUID="f94057a7-f3f0-4b15-99aa-475d330452a1"
Nov 02 12:45:03.553392 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:03.553369    4464 generic.go:356] "Generic (PLEG): container finished" podID="04f90f67-c34e-4f47-8d8f-953837c63533" containerID="de56ff4e9f798ac89a98e3da241d9b22f442a805ac3e13fef76ef2f335bb0eeb" exitCode=60
Nov 02 12:45:03.553504 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:03.553422    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv" event={"ID":"04f90f67-c34e-4f47-8d8f-953837c63533","Type":"ContainerDied","Data":"de56ff4e9f798ac89a98e3da241d9b22f442a805ac3e13fef76ef2f335bb0eeb"}
Nov 02 12:45:03.553619 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:03.553604    4464 scope.go:122] "RemoveContainer" containerID="de56ff4e9f798ac89a98e3da241d9b22f442a805ac3e13fef76ef2f335bb0eeb"
Nov 02 12:45:03.553816 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:45:03.553799    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv_odh-ai-gateway-infra(04f90f67-c34e-4f47-8d8f-953837c63533)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv" podUID="04f90f67-c34e-4f47-8d8f-953837c63533"
Nov 02 12:45:03.562905 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:03.562882    4464 scope.go:122] "RemoveContainer" containerID="b056c3077eeb8ece89185770a4843053946e46271fbab43a8ec6ac3498ff8089"
Nov 02 12:45:03.576683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:03.576658    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:45:04.560542 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:04.560506    4464 scope.go:122] "RemoveContainer" containerID="4dce19c6d13c6476560e5aad6f8c6048fae0b72ad983778c9b5001b6465841d7"
Nov 02 12:45:04.560962 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:45:04.560762    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893725-2vvzv_odh-ai-gateway-infra(f94057a7-f3f0-4b15-99aa-475d330452a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv" podUID="f94057a7-f3f0-4b15-99aa-475d330452a1"
Nov 02 12:45:04.561968 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:04.561952    4464 scope.go:122] "RemoveContainer" containerID="de56ff4e9f798ac89a98e3da241d9b22f442a805ac3e13fef76ef2f335bb0eeb"
Nov 02 12:45:04.562167 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:45:04.562150    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv_odh-ai-gateway-infra(04f90f67-c34e-4f47-8d8f-953837c63533)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv" podUID="04f90f67-c34e-4f47-8d8f-953837c63533"
Nov 02 12:45:13.576094 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:13.576052    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:45:13.946289 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:13.946202    4464 scope.go:122] "RemoveContainer" containerID="02dd95c792a336cd1b0e369264ec628847e654e90f592ffd8c42471f12efb470"
Nov 02 12:45:13.955086 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:13.955065    4464 scope.go:122] "RemoveContainer" containerID="0402b9ca56529f05ebf22e00676cdf6e8a45fa059e8429a4bb6251211c9ea73b"
Nov 02 12:45:13.963061 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:13.962991    4464 scope.go:122] "RemoveContainer" containerID="19417c8ceb12073e2d0e0ea50c12d4121325e154d5cd3ae774a69b529e2b83d2"
Nov 02 12:45:13.970364 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:13.970347    4464 scope.go:122] "RemoveContainer" containerID="0b7e4c1d18d8579b5938562198f800cbe1ca60a562e272e9034f1cc62d21d22e"
Nov 02 12:45:13.977599 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:13.977574    4464 scope.go:122] "RemoveContainer" containerID="d5cc73c125ecb9bcf0fc9e7a9cb6b0586764838d584d8576c14dbc29334b26dd"
Nov 02 12:45:16.575683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:16.575645    4464 scope.go:122] "RemoveContainer" containerID="4dce19c6d13c6476560e5aad6f8c6048fae0b72ad983778c9b5001b6465841d7"
Nov 02 12:45:17.575087 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:17.575006    4464 scope.go:122] "RemoveContainer" containerID="de56ff4e9f798ac89a98e3da241d9b22f442a805ac3e13fef76ef2f335bb0eeb"
Nov 02 12:45:17.633420 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:17.633393    4464 generic.go:356] "Generic (PLEG): container finished" podID="f94057a7-f3f0-4b15-99aa-475d330452a1" containerID="9e9e7c168fa8d59519aaa84cfffdd602723de96e97cc3e3a25b5e3ae8fc73ff6" exitCode=60
Nov 02 12:45:17.633800 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:17.633463    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv" event={"ID":"f94057a7-f3f0-4b15-99aa-475d330452a1","Type":"ContainerDied","Data":"9e9e7c168fa8d59519aaa84cfffdd602723de96e97cc3e3a25b5e3ae8fc73ff6"}
Nov 02 12:45:17.633800 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:17.633511    4464 scope.go:122] "RemoveContainer" containerID="4dce19c6d13c6476560e5aad6f8c6048fae0b72ad983778c9b5001b6465841d7"
Nov 02 12:45:17.633922 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:17.633857    4464 scope.go:122] "RemoveContainer" containerID="9e9e7c168fa8d59519aaa84cfffdd602723de96e97cc3e3a25b5e3ae8fc73ff6"
Nov 02 12:45:17.634119 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:45:17.634098    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893725-2vvzv_odh-ai-gateway-infra(f94057a7-f3f0-4b15-99aa-475d330452a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv" podUID="f94057a7-f3f0-4b15-99aa-475d330452a1"
Nov 02 12:45:18.640252 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:18.640165    4464 generic.go:356] "Generic (PLEG): container finished" podID="04f90f67-c34e-4f47-8d8f-953837c63533" containerID="1ad3a6dd6355deaea1a05fe4488a6e78f3907ec6c5476f3550426c0ca19c5779" exitCode=60
Nov 02 12:45:18.640252 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:18.640199    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv" event={"ID":"04f90f67-c34e-4f47-8d8f-953837c63533","Type":"ContainerDied","Data":"1ad3a6dd6355deaea1a05fe4488a6e78f3907ec6c5476f3550426c0ca19c5779"}
Nov 02 12:45:18.640783 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:18.640265    4464 scope.go:122] "RemoveContainer" containerID="de56ff4e9f798ac89a98e3da241d9b22f442a805ac3e13fef76ef2f335bb0eeb"
Nov 02 12:45:18.640783 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:18.640675    4464 scope.go:122] "RemoveContainer" containerID="1ad3a6dd6355deaea1a05fe4488a6e78f3907ec6c5476f3550426c0ca19c5779"
Nov 02 12:45:18.640999 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:45:18.640962    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv_odh-ai-gateway-infra(04f90f67-c34e-4f47-8d8f-953837c63533)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv" podUID="04f90f67-c34e-4f47-8d8f-953837c63533"
Nov 02 12:45:18.762728 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:18.762693    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv"]
Nov 02 12:45:18.900602 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:18.900543    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv"
Nov 02 12:45:18.964828 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:18.964801    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f94057a7-f3f0-4b15-99aa-475d330452a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94057a7-f3f0-4b15-99aa-475d330452a1-maas-api-service-ca\") pod \"f94057a7-f3f0-4b15-99aa-475d330452a1\" (UID: \"f94057a7-f3f0-4b15-99aa-475d330452a1\") "
Nov 02 12:45:18.964954 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:18.964896    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f94057a7-f3f0-4b15-99aa-475d330452a1-kube-api-access-h6vd2\" (UniqueName: \"kubernetes.io/projected/f94057a7-f3f0-4b15-99aa-475d330452a1-kube-api-access-h6vd2\") pod \"f94057a7-f3f0-4b15-99aa-475d330452a1\" (UID: \"f94057a7-f3f0-4b15-99aa-475d330452a1\") "
Nov 02 12:45:18.965173 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:18.965147    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94057a7-f3f0-4b15-99aa-475d330452a1-maas-api-service-ca" pod "f94057a7-f3f0-4b15-99aa-475d330452a1" (UID: "f94057a7-f3f0-4b15-99aa-475d330452a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:18.966878 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:18.966855    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94057a7-f3f0-4b15-99aa-475d330452a1-kube-api-access-h6vd2" pod "f94057a7-f3f0-4b15-99aa-475d330452a1" (UID: "f94057a7-f3f0-4b15-99aa-475d330452a1"). InnerVolumeSpecName "kube-api-access-h6vd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:19.065949 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:19.065916    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f94057a7-f3f0-4b15-99aa-475d330452a1-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:45:19.065949 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:19.065945    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6vd2\" (UniqueName: \"kubernetes.io/projected/f94057a7-f3f0-4b15-99aa-475d330452a1-kube-api-access-h6vd2\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:45:19.647860 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:19.647828    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv"
Nov 02 12:45:19.647860 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:19.647840    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv" event={"ID":"f94057a7-f3f0-4b15-99aa-475d330452a1","Type":"ContainerDied","Data":"311eaec87c95a3040ab8b98dacfce9b814084f761e672225f2ff70b6e1866fa9"}
Nov 02 12:45:19.648410 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:19.647899    4464 scope.go:122] "RemoveContainer" containerID="9e9e7c168fa8d59519aaa84cfffdd602723de96e97cc3e3a25b5e3ae8fc73ff6"
Nov 02 12:45:19.767277 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:19.767243    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv"]
Nov 02 12:45:19.815581 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:19.815550    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv"]
Nov 02 12:45:19.819964 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:19.819939    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893725-2vvzv"]
Nov 02 12:45:19.908293 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:19.908270    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv"
Nov 02 12:45:19.972396 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:19.972363    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f90f67-c34e-4f47-8d8f-953837c63533-kube-api-access-sffjt\" (UniqueName: \"kubernetes.io/projected/04f90f67-c34e-4f47-8d8f-953837c63533-kube-api-access-sffjt\") pod \"04f90f67-c34e-4f47-8d8f-953837c63533\" (UID: \"04f90f67-c34e-4f47-8d8f-953837c63533\") "
Nov 02 12:45:19.972569 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:19.972473    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04f90f67-c34e-4f47-8d8f-953837c63533-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f90f67-c34e-4f47-8d8f-953837c63533-maas-api-service-ca\") pod \"04f90f67-c34e-4f47-8d8f-953837c63533\" (UID: \"04f90f67-c34e-4f47-8d8f-953837c63533\") "
Nov 02 12:45:19.972806 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:19.972785    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f90f67-c34e-4f47-8d8f-953837c63533-maas-api-service-ca" pod "04f90f67-c34e-4f47-8d8f-953837c63533" (UID: "04f90f67-c34e-4f47-8d8f-953837c63533"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:19.974380 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:19.974361    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f90f67-c34e-4f47-8d8f-953837c63533-kube-api-access-sffjt" pod "04f90f67-c34e-4f47-8d8f-953837c63533" (UID: "04f90f67-c34e-4f47-8d8f-953837c63533"). InnerVolumeSpecName "kube-api-access-sffjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:20.073197 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:20.073164    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sffjt\" (UniqueName: \"kubernetes.io/projected/04f90f67-c34e-4f47-8d8f-953837c63533-kube-api-access-sffjt\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:45:20.073197 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:20.073190    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f90f67-c34e-4f47-8d8f-953837c63533-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:45:20.656081 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:20.656050    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv"
Nov 02 12:45:20.656527 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:20.656088    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv" event={"ID":"04f90f67-c34e-4f47-8d8f-953837c63533","Type":"ContainerDied","Data":"3d3ce42729d301f856420c78c1c6b8ada94056bf9c0d4986051fc87737f560a7"}
Nov 02 12:45:20.656527 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:20.656151    4464 scope.go:122] "RemoveContainer" containerID="1ad3a6dd6355deaea1a05fe4488a6e78f3907ec6c5476f3550426c0ca19c5779"
Nov 02 12:45:20.822892 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:20.822860    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv"]
Nov 02 12:45:20.827158 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:20.827130    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893725-zc6lv"]
Nov 02 12:45:21.580806 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:21.580774    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04f90f67-c34e-4f47-8d8f-953837c63533" path="/var/lib/kubelet/pods/04f90f67-c34e-4f47-8d8f-953837c63533/volumes"
Nov 02 12:45:21.581190 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:21.581176    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f94057a7-f3f0-4b15-99aa-475d330452a1" path="/var/lib/kubelet/pods/f94057a7-f3f0-4b15-99aa-475d330452a1/volumes"
Nov 02 12:45:23.576218 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:23.576181    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:45:28.292027 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:45:28.291966    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:45:33.576233 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:33.576194    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:45:35.057733 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:35.057692    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"]
Nov 02 12:45:35.108278 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:35.108238    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87ba9b34-lxqhd"]
Nov 02 12:45:35.581383 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:35.581346    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c37bdd-7327-4b35-900c-7aa9065c6466" path="/var/lib/kubelet/pods/b2c37bdd-7327-4b35-900c-7aa9065c6466/volumes"
Nov 02 12:45:43.576051 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:43.576011    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:45:44.575992 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:44.575943    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:45:54.575967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:45:54.575926    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:45:58.291483 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:45:58.291448    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:46:04.576670 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:04.576627    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:46:11.987352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:11.987325    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:46:12.005041 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:12.005021    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:46:12.025280 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:12.025259    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:46:12.042532 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:12.042511    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:46:14.011200 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:14.011164    4464 scope.go:122] "RemoveContainer" containerID="309ec739bd4839cdec6953e4c66f68c463236f59e3f0d04e649ff51ebc28ebd6"
Nov 02 12:46:14.576819 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:14.576779    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:46:17.468163 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.468132    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"]
Nov 02 12:46:17.468572 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.468559    4464 cpu_manager.go:414] "containerMap: removing container" podUID="04f90f67-c34e-4f47-8d8f-953837c63533" containerName="cleanup"
Nov 02 12:46:17.468618 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.468572    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f90f67-c34e-4f47-8d8f-953837c63533" containerName="cleanup"
Nov 02 12:46:17.468618 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.468589    4464 cpu_manager.go:414] "containerMap: removing container" podUID="04f90f67-c34e-4f47-8d8f-953837c63533" containerName="cleanup"
Nov 02 12:46:17.468618 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.468594    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f90f67-c34e-4f47-8d8f-953837c63533" containerName="cleanup"
Nov 02 12:46:17.468618 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.468602    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f94057a7-f3f0-4b15-99aa-475d330452a1" containerName="cleanup"
Nov 02 12:46:17.468618 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.468607    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94057a7-f3f0-4b15-99aa-475d330452a1" containerName="cleanup"
Nov 02 12:46:17.468618 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.468619    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f94057a7-f3f0-4b15-99aa-475d330452a1" containerName="cleanup"
Nov 02 12:46:17.468812 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.468624    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94057a7-f3f0-4b15-99aa-475d330452a1" containerName="cleanup"
Nov 02 12:46:17.468812 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.468634    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f94057a7-f3f0-4b15-99aa-475d330452a1" containerName="cleanup"
Nov 02 12:46:17.468812 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.468639    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94057a7-f3f0-4b15-99aa-475d330452a1" containerName="cleanup"
Nov 02 12:46:17.468812 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.468644    4464 cpu_manager.go:414] "containerMap: removing container" podUID="04f90f67-c34e-4f47-8d8f-953837c63533" containerName="cleanup"
Nov 02 12:46:17.468812 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.468648    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f90f67-c34e-4f47-8d8f-953837c63533" containerName="cleanup"
Nov 02 12:46:17.468812 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.468722    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f90f67-c34e-4f47-8d8f-953837c63533" containerName="cleanup"
Nov 02 12:46:17.468812 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.468730    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94057a7-f3f0-4b15-99aa-475d330452a1" containerName="cleanup"
Nov 02 12:46:17.468812 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.468736    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94057a7-f3f0-4b15-99aa-475d330452a1" containerName="cleanup"
Nov 02 12:46:17.468812 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.468742    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f90f67-c34e-4f47-8d8f-953837c63533" containerName="cleanup"
Nov 02 12:46:17.472052 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.472036    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"
Nov 02 12:46:17.527351 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.527324    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:46:17.577966 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.577936    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:46:17.578170 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.578092    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:46:17.578170 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.578115    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:46:17.580764 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.580743    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"]
Nov 02 12:46:17.623293 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.623269    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7acb3ce-jwjng\" (UID: \"f1de5a10-9d79-4eda-b4c2-36f2f923bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"
Nov 02 12:46:17.623381 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.623301    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bdj6\" (UniqueName: \"kubernetes.io/projected/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-kube-api-access-8bdj6\") pod \"maas-api-revoke-sub-d7acb3ce-jwjng\" (UID: \"f1de5a10-9d79-4eda-b4c2-36f2f923bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"
Nov 02 12:46:17.623435 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.623418    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7acb3ce-jwjng\" (UID: \"f1de5a10-9d79-4eda-b4c2-36f2f923bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"
Nov 02 12:46:17.724733 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.724709    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7acb3ce-jwjng\" (UID: \"f1de5a10-9d79-4eda-b4c2-36f2f923bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"
Nov 02 12:46:17.724890 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.724743    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bdj6\" (UniqueName: \"kubernetes.io/projected/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-kube-api-access-8bdj6\") pod \"maas-api-revoke-sub-d7acb3ce-jwjng\" (UID: \"f1de5a10-9d79-4eda-b4c2-36f2f923bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"
Nov 02 12:46:17.724890 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.724800    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7acb3ce-jwjng\" (UID: \"f1de5a10-9d79-4eda-b4c2-36f2f923bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"
Nov 02 12:46:17.725413 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.725387    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7acb3ce-jwjng\" (UID: \"f1de5a10-9d79-4eda-b4c2-36f2f923bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"
Nov 02 12:46:17.727009 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.726988    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7acb3ce-jwjng\" (UID: \"f1de5a10-9d79-4eda-b4c2-36f2f923bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"
Nov 02 12:46:17.786362 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:17.786336    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bdj6\" (UniqueName: \"kubernetes.io/projected/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-kube-api-access-8bdj6\") pod \"maas-api-revoke-sub-d7acb3ce-jwjng\" (UID: \"f1de5a10-9d79-4eda-b4c2-36f2f923bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"
Nov 02 12:46:18.084359 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:18.084282    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"
Nov 02 12:46:18.454940 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:18.454910    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"]
Nov 02 12:46:18.456467 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:46:18.456443    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1de5a10_9d79_4eda_b4c2_36f2f923bc96.slice/crio-aa4b127c2a4f950d230e85f9649dc43fe6c8ad3fa4d47eeee1af5ec283d2588f WatchSource:0}: Error finding container aa4b127c2a4f950d230e85f9649dc43fe6c8ad3fa4d47eeee1af5ec283d2588f: Status 404 returned error can't find the container with id aa4b127c2a4f950d230e85f9649dc43fe6c8ad3fa4d47eeee1af5ec283d2588f
Nov 02 12:46:18.946267 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:18.946235    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng" event={"ID":"f1de5a10-9d79-4eda-b4c2-36f2f923bc96","Type":"ContainerStarted","Data":"e338507f26a9204381778f97bbae2e4d67c927a46c996df8bd7360f0686910be"}
Nov 02 12:46:18.946689 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:18.946290    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng" event={"ID":"f1de5a10-9d79-4eda-b4c2-36f2f923bc96","Type":"ContainerStarted","Data":"aa4b127c2a4f950d230e85f9649dc43fe6c8ad3fa4d47eeee1af5ec283d2588f"}
Nov 02 12:46:19.070239 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:19.070143    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng" podStartSLOduration=2.070128069 podStartE2EDuration="2.070128069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:19.06829168 +0000 UTC m=+3007.632000090" watchObservedRunningTime="2026-11-02 12:46:19.070128069 +0000 UTC m=+3007.633836477"
Nov 02 12:46:19.952635 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:19.952601    4464 generic.go:356] "Generic (PLEG): container finished" podID="f1de5a10-9d79-4eda-b4c2-36f2f923bc96" containerID="e338507f26a9204381778f97bbae2e4d67c927a46c996df8bd7360f0686910be" exitCode=0
Nov 02 12:46:19.953045 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:19.952682    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng" event={"ID":"f1de5a10-9d79-4eda-b4c2-36f2f923bc96","Type":"ContainerDied","Data":"e338507f26a9204381778f97bbae2e4d67c927a46c996df8bd7360f0686910be"}
Nov 02 12:46:21.089034 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:21.089013    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"
Nov 02 12:46:21.159711 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:21.159687    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-maas-api-xks-ca\") pod \"f1de5a10-9d79-4eda-b4c2-36f2f923bc96\" (UID: \"f1de5a10-9d79-4eda-b4c2-36f2f923bc96\") "
Nov 02 12:46:21.159840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:21.159744    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-kube-api-access-8bdj6\" (UniqueName: \"kubernetes.io/projected/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-kube-api-access-8bdj6\") pod \"f1de5a10-9d79-4eda-b4c2-36f2f923bc96\" (UID: \"f1de5a10-9d79-4eda-b4c2-36f2f923bc96\") "
Nov 02 12:46:21.159840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:21.159775    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-maas-api-openshift-ca\") pod \"f1de5a10-9d79-4eda-b4c2-36f2f923bc96\" (UID: \"f1de5a10-9d79-4eda-b4c2-36f2f923bc96\") "
Nov 02 12:46:21.160196 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:21.160174    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-maas-api-openshift-ca" pod "f1de5a10-9d79-4eda-b4c2-36f2f923bc96" (UID: "f1de5a10-9d79-4eda-b4c2-36f2f923bc96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:21.161840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:21.161818    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-maas-api-xks-ca" pod "f1de5a10-9d79-4eda-b4c2-36f2f923bc96" (UID: "f1de5a10-9d79-4eda-b4c2-36f2f923bc96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:46:21.162011 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:21.161988    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-kube-api-access-8bdj6" pod "f1de5a10-9d79-4eda-b4c2-36f2f923bc96" (UID: "f1de5a10-9d79-4eda-b4c2-36f2f923bc96"). InnerVolumeSpecName "kube-api-access-8bdj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:21.261518 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:21.261492    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:46:21.261518 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:21.261515    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bdj6\" (UniqueName: \"kubernetes.io/projected/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-kube-api-access-8bdj6\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:46:21.261690 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:21.261526    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1de5a10-9d79-4eda-b4c2-36f2f923bc96-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:46:21.963333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:21.963260    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"
Nov 02 12:46:21.963333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:21.963293    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng" event={"ID":"f1de5a10-9d79-4eda-b4c2-36f2f923bc96","Type":"ContainerDied","Data":"aa4b127c2a4f950d230e85f9649dc43fe6c8ad3fa4d47eeee1af5ec283d2588f"}
Nov 02 12:46:21.963524 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:21.963338    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa4b127c2a4f950d230e85f9649dc43fe6c8ad3fa4d47eeee1af5ec283d2588f"
Nov 02 12:46:24.576552 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:24.576512    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:46:28.291402 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:46:28.291360    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:46:29.676176 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:29.676141    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-extmod-a-6ac03f_payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g_62f830a9-2aa4-4720-a5a8-3ceb98d69c3a/payload-processing/0.log"
Nov 02 12:46:34.306135 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:34.306095    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:46:34.576098 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:34.575996    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:46:35.192933 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.192896    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"]
Nov 02 12:46:35.193210 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:46:35.193191    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv" podUID="242becf2-522b-492b-b390-e56675435069"
Nov 02 12:46:35.228518 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.228482    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"]
Nov 02 12:46:35.229085 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.229059    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f1de5a10-9d79-4eda-b4c2-36f2f923bc96" containerName="revoke-keys"
Nov 02 12:46:35.229085 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.229078    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1de5a10-9d79-4eda-b4c2-36f2f923bc96" containerName="revoke-keys"
Nov 02 12:46:35.229280 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.229171    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1de5a10-9d79-4eda-b4c2-36f2f923bc96" containerName="revoke-keys"
Nov 02 12:46:35.229280 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.229179    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94057a7-f3f0-4b15-99aa-475d330452a1" containerName="cleanup"
Nov 02 12:46:35.229280 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.229193    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f90f67-c34e-4f47-8d8f-953837c63533" containerName="cleanup"
Nov 02 12:46:35.232650 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.232634    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:46:35.285716 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.285687    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"agc-cred-d30a62f29ed0\""
Nov 02 12:46:35.302792 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.302758    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1463f044-a7fd-490d-a616-2bcc2c540305-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk\" (UID: \"1463f044-a7fd-490d-a616-2bcc2c540305\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:46:35.302967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.302865    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1463f044-a7fd-490d-a616-2bcc2c540305-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk\" (UID: \"1463f044-a7fd-490d-a616-2bcc2c540305\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:46:35.302967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.302874    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"]
Nov 02 12:46:35.302967 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.302944    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9nhq\" (UniqueName: \"kubernetes.io/projected/1463f044-a7fd-490d-a616-2bcc2c540305-kube-api-access-b9nhq\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk\" (UID: \"1463f044-a7fd-490d-a616-2bcc2c540305\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:46:35.303118 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.303048    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1463f044-a7fd-490d-a616-2bcc2c540305-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk\" (UID: \"1463f044-a7fd-490d-a616-2bcc2c540305\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:46:35.404411 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.404375    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1463f044-a7fd-490d-a616-2bcc2c540305-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk\" (UID: \"1463f044-a7fd-490d-a616-2bcc2c540305\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:46:35.404789 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.404419    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1463f044-a7fd-490d-a616-2bcc2c540305-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk\" (UID: \"1463f044-a7fd-490d-a616-2bcc2c540305\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:46:35.404789 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.404439    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9nhq\" (UniqueName: \"kubernetes.io/projected/1463f044-a7fd-490d-a616-2bcc2c540305-kube-api-access-b9nhq\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk\" (UID: \"1463f044-a7fd-490d-a616-2bcc2c540305\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:46:35.404789 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.404551    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1463f044-a7fd-490d-a616-2bcc2c540305-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk\" (UID: \"1463f044-a7fd-490d-a616-2bcc2c540305\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:46:35.405298 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.405268    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1463f044-a7fd-490d-a616-2bcc2c540305-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk\" (UID: \"1463f044-a7fd-490d-a616-2bcc2c540305\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:46:35.405422 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.405396    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1463f044-a7fd-490d-a616-2bcc2c540305-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk\" (UID: \"1463f044-a7fd-490d-a616-2bcc2c540305\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:46:35.407021 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.406994    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1463f044-a7fd-490d-a616-2bcc2c540305-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk\" (UID: \"1463f044-a7fd-490d-a616-2bcc2c540305\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:46:35.466507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.466435    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9nhq\" (UniqueName: \"kubernetes.io/projected/1463f044-a7fd-490d-a616-2bcc2c540305-kube-api-access-b9nhq\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk\" (UID: \"1463f044-a7fd-490d-a616-2bcc2c540305\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:46:35.545897 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.545867    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:46:35.918729 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:35.918699    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"]
Nov 02 12:46:35.920562 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:46:35.920539    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1463f044_a7fd_490d_a616_2bcc2c540305.slice/crio-9bb3a41b83194bc86bbc2aca6527214a6d334db1c56e090a5f6bbb16b2492d47 WatchSource:0}: Error finding container 9bb3a41b83194bc86bbc2aca6527214a6d334db1c56e090a5f6bbb16b2492d47: Status 404 returned error can't find the container with id 9bb3a41b83194bc86bbc2aca6527214a6d334db1c56e090a5f6bbb16b2492d47
Nov 02 12:46:36.035035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:36.035004    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:46:36.035035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:36.035019    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk" event={"ID":"1463f044-a7fd-490d-a616-2bcc2c540305","Type":"ContainerStarted","Data":"8f3dfacc2e4df437a71bf89cf4a45672aaec09090f3058b6ecb9a5447c70ee97"}
Nov 02 12:46:36.035244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:36.035069    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk" event={"ID":"1463f044-a7fd-490d-a616-2bcc2c540305","Type":"ContainerStarted","Data":"9bb3a41b83194bc86bbc2aca6527214a6d334db1c56e090a5f6bbb16b2492d47"}
Nov 02 12:46:36.035244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:36.035097    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:46:36.041610 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:36.041591    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:46:36.111295 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:36.111269    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-kube-api-access-kr5cq\" (UniqueName: \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-kube-api-access-kr5cq\") pod \"242becf2-522b-492b-b390-e56675435069\" (UID: \"242becf2-522b-492b-b390-e56675435069\") "
Nov 02 12:46:36.111435 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:36.111366    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/242becf2-522b-492b-b390-e56675435069-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/242becf2-522b-492b-b390-e56675435069-routing-overlay-volume\") pod \"242becf2-522b-492b-b390-e56675435069\" (UID: \"242becf2-522b-492b-b390-e56675435069\") "
Nov 02 12:46:36.111531 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:36.111517    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/242becf2-522b-492b-b390-e56675435069-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/242becf2-522b-492b-b390-e56675435069-plugins-config-volume\") pod \"242becf2-522b-492b-b390-e56675435069\" (UID: \"242becf2-522b-492b-b390-e56675435069\") "
Nov 02 12:46:36.111694 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:36.111672    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242becf2-522b-492b-b390-e56675435069-routing-overlay-volume" pod "242becf2-522b-492b-b390-e56675435069" (UID: "242becf2-522b-492b-b390-e56675435069"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:36.111922 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:36.111898    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242becf2-522b-492b-b390-e56675435069-plugins-config-volume" pod "242becf2-522b-492b-b390-e56675435069" (UID: "242becf2-522b-492b-b390-e56675435069"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:36.112614 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:36.111958    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/242becf2-522b-492b-b390-e56675435069-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:46:36.113575 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:36.113555    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-kube-api-access-kr5cq" pod "242becf2-522b-492b-b390-e56675435069" (UID: "242becf2-522b-492b-b390-e56675435069"). InnerVolumeSpecName "kube-api-access-kr5cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:36.154916 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:36.154870    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk" podStartSLOduration=1.154860064 podStartE2EDuration="1.154860064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:36.15429301 +0000 UTC m=+3024.718001421" watchObservedRunningTime="2026-11-02 12:46:36.154860064 +0000 UTC m=+3024.718568473"
Nov 02 12:46:36.212925 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:36.212855    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/242becf2-522b-492b-b390-e56675435069-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:46:36.212925 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:36.212883    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr5cq\" (UniqueName: \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-kube-api-access-kr5cq\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:46:37.039814 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:37.039780    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"
Nov 02 12:46:37.270175 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:37.270143    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"]
Nov 02 12:46:37.274774 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:37.274749    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-95776q6lv"]
Nov 02 12:46:37.323994 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:37.323915    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/242becf2-522b-492b-b390-e56675435069-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:46:37.581099 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:37.581025    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="242becf2-522b-492b-b390-e56675435069" path="/var/lib/kubelet/pods/242becf2-522b-492b-b390-e56675435069/volumes"
Nov 02 12:46:44.576075 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:44.576040    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:46:47.042971 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:47.042938    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:46:47.220287 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:47.220248    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"]
Nov 02 12:46:47.221794 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:47.221759    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:46:52.221115 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:52.221045    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing" containerID="cri-o://f423f7b0c5060cebe5107bc8db224898181062300f8c706772668f715d053581" gracePeriod=25
Nov 02 12:46:52.474327 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:52.474268    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"
Nov 02 12:46:52.564902 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:52.564868    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-plugins-config-volume\") pod \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\" (UID: \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\") "
Nov 02 12:46:52.565070 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:52.564913    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-provider-credentials\") pod \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\" (UID: \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\") "
Nov 02 12:46:52.565070 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:52.564929    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-kube-api-access-675cc\" (UniqueName: \"kubernetes.io/projected/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-kube-api-access-675cc\") pod \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\" (UID: \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\") "
Nov 02 12:46:52.565201 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:52.565114    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-routing-overlay-volume\") pod \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\" (UID: \"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a\") "
Nov 02 12:46:52.565312 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:52.565290    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-plugins-config-volume" pod "62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" (UID: "62f830a9-2aa4-4720-a5a8-3ceb98d69c3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:52.565481 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:52.565459    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-routing-overlay-volume" pod "62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" (UID: "62f830a9-2aa4-4720-a5a8-3ceb98d69c3a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:52.565635 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:52.565619    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:46:52.565691 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:52.565641    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:46:52.567147 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:52.567122    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-kube-api-access-675cc" pod "62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" (UID: "62f830a9-2aa4-4720-a5a8-3ceb98d69c3a"). InnerVolumeSpecName "kube-api-access-675cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:52.567147 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:52.567130    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-provider-credentials" pod "62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" (UID: "62f830a9-2aa4-4720-a5a8-3ceb98d69c3a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:52.666220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:52.666191    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:46:52.666220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:52.666215    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-675cc\" (UniqueName: \"kubernetes.io/projected/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a-kube-api-access-675cc\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:46:53.124473 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:53.124435    4464 generic.go:356] "Generic (PLEG): container finished" podID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerID="f423f7b0c5060cebe5107bc8db224898181062300f8c706772668f715d053581" exitCode=0
Nov 02 12:46:53.124657 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:53.124483    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"
Nov 02 12:46:53.124657 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:53.124513    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" event={"ID":"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a","Type":"ContainerDied","Data":"f423f7b0c5060cebe5107bc8db224898181062300f8c706772668f715d053581"}
Nov 02 12:46:53.124657 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:53.124568    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g" event={"ID":"62f830a9-2aa4-4720-a5a8-3ceb98d69c3a","Type":"ContainerDied","Data":"d91f3bf274c441a3f0b63579bce23486913e714a480c83ff3a8fd81e4f8dcd7c"}
Nov 02 12:46:53.124657 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:53.124591    4464 scope.go:122] "RemoveContainer" containerID="f423f7b0c5060cebe5107bc8db224898181062300f8c706772668f715d053581"
Nov 02 12:46:53.133824 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:53.133808    4464 scope.go:122] "RemoveContainer" containerID="f423f7b0c5060cebe5107bc8db224898181062300f8c706772668f715d053581"
Nov 02 12:46:53.134108 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:46:53.134086    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f423f7b0c5060cebe5107bc8db224898181062300f8c706772668f715d053581\": container with ID starting with f423f7b0c5060cebe5107bc8db224898181062300f8c706772668f715d053581 not found: ID does not exist" containerID="f423f7b0c5060cebe5107bc8db224898181062300f8c706772668f715d053581"
Nov 02 12:46:53.134173 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:53.134119    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f423f7b0c5060cebe5107bc8db224898181062300f8c706772668f715d053581"} err="failed to get container status \"f423f7b0c5060cebe5107bc8db224898181062300f8c706772668f715d053581\": rpc error: code = NotFound desc = could not find container \"f423f7b0c5060cebe5107bc8db224898181062300f8c706772668f715d053581\": container with ID starting with f423f7b0c5060cebe5107bc8db224898181062300f8c706772668f715d053581 not found: ID does not exist"
Nov 02 12:46:53.291797 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:53.291762    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"]
Nov 02 12:46:53.296073 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:53.296050    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6c7dwgj6g"]
Nov 02 12:46:53.580825 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:46:53.580792    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" path="/var/lib/kubelet/pods/62f830a9-2aa4-4720-a5a8-3ceb98d69c3a/volumes"
Nov 02 12:46:58.291672 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:46:58.291635    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:47:03.607763 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:47:03.607724    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:47:14.039814 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:47:14.039776    4464 scope.go:122] "RemoveContainer" containerID="2a8c90006099a0f405b86bc1f178c2efb1ef70da8be6145eb20fdc1add84f800"
Nov 02 12:47:28.292532 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:47:28.292427    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:47:58.292129 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:47:58.292051    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:48:28.290585 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:48:28.290549    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:48:58.292362 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:48:58.292322    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:49:07.093769 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:07.093737    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-extmod-a-6ac03f_payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk_1463f044-a7fd-490d-a616-2bcc2c540305/payload-processing/0.log"
Nov 02 12:49:12.473668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.473631    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"]
Nov 02 12:49:12.474123 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.474109    4464 cpu_manager.go:414] "containerMap: removing container" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing"
Nov 02 12:49:12.474188 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.474124    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing"
Nov 02 12:49:12.474228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.474210    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f830a9-2aa4-4720-a5a8-3ceb98d69c3a" containerName="payload-processing"
Nov 02 12:49:12.477457 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.477430    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"
Nov 02 12:49:12.530534 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.530501    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"agc-cred-c501215b3849\""
Nov 02 12:49:12.581421 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.581386    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r\" (UID: \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"
Nov 02 12:49:12.581604 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.581441    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r\" (UID: \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"
Nov 02 12:49:12.581604 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.581557    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rptr\" (UniqueName: \"kubernetes.io/projected/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-kube-api-access-6rptr\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r\" (UID: \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"
Nov 02 12:49:12.581731 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.581668    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r\" (UID: \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"
Nov 02 12:49:12.583206 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.583174    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"]
Nov 02 12:49:12.683263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.683228    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r\" (UID: \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"
Nov 02 12:49:12.683497 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.683285    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r\" (UID: \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"
Nov 02 12:49:12.683497 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.683319    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rptr\" (UniqueName: \"kubernetes.io/projected/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-kube-api-access-6rptr\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r\" (UID: \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"
Nov 02 12:49:12.683497 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.683373    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r\" (UID: \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"
Nov 02 12:49:12.684024 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.683999    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r\" (UID: \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"
Nov 02 12:49:12.684161 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.684136    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r\" (UID: \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"
Nov 02 12:49:12.685841 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.685815    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r\" (UID: \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"
Nov 02 12:49:12.745443 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.745374    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rptr\" (UniqueName: \"kubernetes.io/projected/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-kube-api-access-6rptr\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r\" (UID: \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"
Nov 02 12:49:12.790847 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:12.790802    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"
Nov 02 12:49:13.167357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:13.167331    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"]
Nov 02 12:49:13.169057 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:49:13.169029    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7cb7aad_d583_40c7_ab6e_7ce37c712cb6.slice/crio-e7e26a07bf98b8240d7564f90488d61ca66fe39eb7cb283f3ded037d29dc447d WatchSource:0}: Error finding container e7e26a07bf98b8240d7564f90488d61ca66fe39eb7cb283f3ded037d29dc447d: Status 404 returned error can't find the container with id e7e26a07bf98b8240d7564f90488d61ca66fe39eb7cb283f3ded037d29dc447d
Nov 02 12:49:13.811643 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:13.811602    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r" event={"ID":"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6","Type":"ContainerStarted","Data":"32b66c466ae4faaa07d8be58d4c3dc642c5d55352abe55f2c2694fa6776cdd6a"}
Nov 02 12:49:13.812063 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:13.811662    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r" event={"ID":"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6","Type":"ContainerStarted","Data":"e7e26a07bf98b8240d7564f90488d61ca66fe39eb7cb283f3ded037d29dc447d"}
Nov 02 12:49:13.812063 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:13.811692    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"
Nov 02 12:49:13.930240 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:13.930186    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r" podStartSLOduration=1.930172607 podStartE2EDuration="1.930172607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:13.929775539 +0000 UTC m=+3182.493483948" watchObservedRunningTime="2026-11-02 12:49:13.930172607 +0000 UTC m=+3182.493881016"
Nov 02 12:49:14.571017 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:14.570961    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:49:15.064685 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.064639    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"]
Nov 02 12:49:15.115190 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.115156    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"]
Nov 02 12:49:15.119213 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.119194    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:49:15.174366 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.174333    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"agc-cred-bbd2f8b42d89\""
Nov 02 12:49:15.177227 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.177203    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"]
Nov 02 12:49:15.310635 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.310591    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bnj6\" (UniqueName: \"kubernetes.io/projected/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-kube-api-access-5bnj6\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz\" (UID: \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:49:15.310817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.310704    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz\" (UID: \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:49:15.310817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.310747    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz\" (UID: \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:49:15.310912 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.310817    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz\" (UID: \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:49:15.411671 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.411584    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz\" (UID: \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:49:15.411671 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.411627    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz\" (UID: \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:49:15.411946 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.411706    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bnj6\" (UniqueName: \"kubernetes.io/projected/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-kube-api-access-5bnj6\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz\" (UID: \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:49:15.411946 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.411764    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz\" (UID: \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:49:15.412332 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.412307    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz\" (UID: \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:49:15.412514 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.412496    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz\" (UID: \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:49:15.414316 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.414300    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz\" (UID: \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:49:15.473626 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.473592    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bnj6\" (UniqueName: \"kubernetes.io/projected/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-kube-api-access-5bnj6\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz\" (UID: \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:49:15.733345 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:15.733297    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:49:16.104959 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:16.104933    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"]
Nov 02 12:49:16.106285 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:49:16.106256    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e301e8f_5107_47d9_8ffc_0b99b9e5d1b0.slice/crio-e3108ebe6fea22061af4f33d96557e6f57349a3e33baabc5f1c03a9ca327bf21 WatchSource:0}: Error finding container e3108ebe6fea22061af4f33d96557e6f57349a3e33baabc5f1c03a9ca327bf21: Status 404 returned error can't find the container with id e3108ebe6fea22061af4f33d96557e6f57349a3e33baabc5f1c03a9ca327bf21
Nov 02 12:49:16.830455 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:16.830424    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz" event={"ID":"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0","Type":"ContainerStarted","Data":"874333aafdb5e979b42d4dede2d3920ae0de43af933e0fcc281669c0d3e0c127"}
Nov 02 12:49:16.830683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:16.830481    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz" event={"ID":"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0","Type":"ContainerStarted","Data":"e3108ebe6fea22061af4f33d96557e6f57349a3e33baabc5f1c03a9ca327bf21"}
Nov 02 12:49:16.830683 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:16.830559    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:49:16.951218 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:16.951156    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz" podStartSLOduration=1.95114304 podStartE2EDuration="1.95114304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:16.949718123 +0000 UTC m=+3185.513426533" watchObservedRunningTime="2026-11-02 12:49:16.95114304 +0000 UTC m=+3185.514851485"
Nov 02 12:49:20.824516 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:20.824443    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r" podUID="f7cb7aad-d583-40c7-ab6e-7ce37c712cb6" containerName="payload-processing" containerID="cri-o://32b66c466ae4faaa07d8be58d4c3dc642c5d55352abe55f2c2694fa6776cdd6a" gracePeriod=25
Nov 02 12:49:21.075704 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.075640    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"
Nov 02 12:49:21.161276 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.161245    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-provider-credentials\") pod \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\" (UID: \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\") "
Nov 02 12:49:21.161480 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.161349    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-kube-api-access-6rptr\" (UniqueName: \"kubernetes.io/projected/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-kube-api-access-6rptr\") pod \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\" (UID: \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\") "
Nov 02 12:49:21.161480 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.161388    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-plugins-config-volume\") pod \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\" (UID: \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\") "
Nov 02 12:49:21.161480 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.161429    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-routing-overlay-volume\") pod \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\" (UID: \"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6\") "
Nov 02 12:49:21.161774 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.161747    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-plugins-config-volume" pod "f7cb7aad-d583-40c7-ab6e-7ce37c712cb6" (UID: "f7cb7aad-d583-40c7-ab6e-7ce37c712cb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:49:21.161854 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.161776    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-routing-overlay-volume" pod "f7cb7aad-d583-40c7-ab6e-7ce37c712cb6" (UID: "f7cb7aad-d583-40c7-ab6e-7ce37c712cb6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:49:21.163294 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.163271    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-provider-credentials" pod "f7cb7aad-d583-40c7-ab6e-7ce37c712cb6" (UID: "f7cb7aad-d583-40c7-ab6e-7ce37c712cb6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:49:21.163408 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.163388    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-kube-api-access-6rptr" pod "f7cb7aad-d583-40c7-ab6e-7ce37c712cb6" (UID: "f7cb7aad-d583-40c7-ab6e-7ce37c712cb6"). InnerVolumeSpecName "kube-api-access-6rptr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:49:21.262299 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.262262    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rptr\" (UniqueName: \"kubernetes.io/projected/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-kube-api-access-6rptr\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:49:21.262299 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.262291    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:49:21.262299 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.262301    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:49:21.262531 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.262310    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:49:21.856360 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.856278    4464 generic.go:356] "Generic (PLEG): container finished" podID="f7cb7aad-d583-40c7-ab6e-7ce37c712cb6" containerID="32b66c466ae4faaa07d8be58d4c3dc642c5d55352abe55f2c2694fa6776cdd6a" exitCode=0
Nov 02 12:49:21.856360 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.856323    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"
Nov 02 12:49:21.856815 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.856360    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r" event={"ID":"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6","Type":"ContainerDied","Data":"32b66c466ae4faaa07d8be58d4c3dc642c5d55352abe55f2c2694fa6776cdd6a"}
Nov 02 12:49:21.856815 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.856411    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r" event={"ID":"f7cb7aad-d583-40c7-ab6e-7ce37c712cb6","Type":"ContainerDied","Data":"e7e26a07bf98b8240d7564f90488d61ca66fe39eb7cb283f3ded037d29dc447d"}
Nov 02 12:49:21.856815 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.856433    4464 scope.go:122] "RemoveContainer" containerID="32b66c466ae4faaa07d8be58d4c3dc642c5d55352abe55f2c2694fa6776cdd6a"
Nov 02 12:49:21.866020 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.866002    4464 scope.go:122] "RemoveContainer" containerID="32b66c466ae4faaa07d8be58d4c3dc642c5d55352abe55f2c2694fa6776cdd6a"
Nov 02 12:49:21.866313 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:49:21.866293    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b66c466ae4faaa07d8be58d4c3dc642c5d55352abe55f2c2694fa6776cdd6a\": container with ID starting with 32b66c466ae4faaa07d8be58d4c3dc642c5d55352abe55f2c2694fa6776cdd6a not found: ID does not exist" containerID="32b66c466ae4faaa07d8be58d4c3dc642c5d55352abe55f2c2694fa6776cdd6a"
Nov 02 12:49:21.866382 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:21.866320    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b66c466ae4faaa07d8be58d4c3dc642c5d55352abe55f2c2694fa6776cdd6a"} err="failed to get container status \"32b66c466ae4faaa07d8be58d4c3dc642c5d55352abe55f2c2694fa6776cdd6a\": rpc error: code = NotFound desc = could not find container \"32b66c466ae4faaa07d8be58d4c3dc642c5d55352abe55f2c2694fa6776cdd6a\": container with ID starting with 32b66c466ae4faaa07d8be58d4c3dc642c5d55352abe55f2c2694fa6776cdd6a not found: ID does not exist"
Nov 02 12:49:22.022651 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:22.022607    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"]
Nov 02 12:49:22.073193 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:22.073162    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b798h9r"]
Nov 02 12:49:23.581222 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:23.581192    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7cb7aad-d583-40c7-ab6e-7ce37c712cb6" path="/var/lib/kubelet/pods/f7cb7aad-d583-40c7-ab6e-7ce37c712cb6/volumes"
Nov 02 12:49:27.839082 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:27.839012    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:49:28.011949 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:28.011912    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"]
Nov 02 12:49:28.292157 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:49:28.292122    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:49:33.013602 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.013563    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk" podUID="1463f044-a7fd-490d-a616-2bcc2c540305" containerName="payload-processing" containerID="cri-o://8f3dfacc2e4df437a71bf89cf4a45672aaec09090f3058b6ecb9a5447c70ee97" gracePeriod=25
Nov 02 12:49:33.262507 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.262476    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:49:33.376949 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.376886    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1463f044-a7fd-490d-a616-2bcc2c540305-kube-api-access-b9nhq\" (UniqueName: \"kubernetes.io/projected/1463f044-a7fd-490d-a616-2bcc2c540305-kube-api-access-b9nhq\") pod \"1463f044-a7fd-490d-a616-2bcc2c540305\" (UID: \"1463f044-a7fd-490d-a616-2bcc2c540305\") "
Nov 02 12:49:33.376949 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.376938    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1463f044-a7fd-490d-a616-2bcc2c540305-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1463f044-a7fd-490d-a616-2bcc2c540305-plugins-config-volume\") pod \"1463f044-a7fd-490d-a616-2bcc2c540305\" (UID: \"1463f044-a7fd-490d-a616-2bcc2c540305\") "
Nov 02 12:49:33.377180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.376990    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1463f044-a7fd-490d-a616-2bcc2c540305-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1463f044-a7fd-490d-a616-2bcc2c540305-routing-overlay-volume\") pod \"1463f044-a7fd-490d-a616-2bcc2c540305\" (UID: \"1463f044-a7fd-490d-a616-2bcc2c540305\") "
Nov 02 12:49:33.377180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.377095    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1463f044-a7fd-490d-a616-2bcc2c540305-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1463f044-a7fd-490d-a616-2bcc2c540305-provider-credentials\") pod \"1463f044-a7fd-490d-a616-2bcc2c540305\" (UID: \"1463f044-a7fd-490d-a616-2bcc2c540305\") "
Nov 02 12:49:33.377413 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.377388    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1463f044-a7fd-490d-a616-2bcc2c540305-plugins-config-volume" pod "1463f044-a7fd-490d-a616-2bcc2c540305" (UID: "1463f044-a7fd-490d-a616-2bcc2c540305"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:49:33.377488 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.377394    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1463f044-a7fd-490d-a616-2bcc2c540305-routing-overlay-volume" pod "1463f044-a7fd-490d-a616-2bcc2c540305" (UID: "1463f044-a7fd-490d-a616-2bcc2c540305"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:49:33.379066 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.379045    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1463f044-a7fd-490d-a616-2bcc2c540305-kube-api-access-b9nhq" pod "1463f044-a7fd-490d-a616-2bcc2c540305" (UID: "1463f044-a7fd-490d-a616-2bcc2c540305"). InnerVolumeSpecName "kube-api-access-b9nhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:49:33.379169 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.379075    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1463f044-a7fd-490d-a616-2bcc2c540305-provider-credentials" pod "1463f044-a7fd-490d-a616-2bcc2c540305" (UID: "1463f044-a7fd-490d-a616-2bcc2c540305"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:49:33.478786 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.478760    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1463f044-a7fd-490d-a616-2bcc2c540305-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:49:33.478786 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.478786    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9nhq\" (UniqueName: \"kubernetes.io/projected/1463f044-a7fd-490d-a616-2bcc2c540305-kube-api-access-b9nhq\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:49:33.478956 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.478795    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1463f044-a7fd-490d-a616-2bcc2c540305-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:49:33.478956 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.478804    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1463f044-a7fd-490d-a616-2bcc2c540305-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:49:33.917550 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.917513    4464 generic.go:356] "Generic (PLEG): container finished" podID="1463f044-a7fd-490d-a616-2bcc2c540305" containerID="8f3dfacc2e4df437a71bf89cf4a45672aaec09090f3058b6ecb9a5447c70ee97" exitCode=0
Nov 02 12:49:33.917734 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.917562    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"
Nov 02 12:49:33.917734 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.917603    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk" event={"ID":"1463f044-a7fd-490d-a616-2bcc2c540305","Type":"ContainerDied","Data":"8f3dfacc2e4df437a71bf89cf4a45672aaec09090f3058b6ecb9a5447c70ee97"}
Nov 02 12:49:33.917734 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.917663    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk" event={"ID":"1463f044-a7fd-490d-a616-2bcc2c540305","Type":"ContainerDied","Data":"9bb3a41b83194bc86bbc2aca6527214a6d334db1c56e090a5f6bbb16b2492d47"}
Nov 02 12:49:33.917734 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.917686    4464 scope.go:122] "RemoveContainer" containerID="8f3dfacc2e4df437a71bf89cf4a45672aaec09090f3058b6ecb9a5447c70ee97"
Nov 02 12:49:33.926535 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.926519    4464 scope.go:122] "RemoveContainer" containerID="8f3dfacc2e4df437a71bf89cf4a45672aaec09090f3058b6ecb9a5447c70ee97"
Nov 02 12:49:33.926814 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:49:33.926792    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3dfacc2e4df437a71bf89cf4a45672aaec09090f3058b6ecb9a5447c70ee97\": container with ID starting with 8f3dfacc2e4df437a71bf89cf4a45672aaec09090f3058b6ecb9a5447c70ee97 not found: ID does not exist" containerID="8f3dfacc2e4df437a71bf89cf4a45672aaec09090f3058b6ecb9a5447c70ee97"
Nov 02 12:49:33.926894 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:33.926823    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3dfacc2e4df437a71bf89cf4a45672aaec09090f3058b6ecb9a5447c70ee97"} err="failed to get container status \"8f3dfacc2e4df437a71bf89cf4a45672aaec09090f3058b6ecb9a5447c70ee97\": rpc error: code = NotFound desc = could not find container \"8f3dfacc2e4df437a71bf89cf4a45672aaec09090f3058b6ecb9a5447c70ee97\": container with ID starting with 8f3dfacc2e4df437a71bf89cf4a45672aaec09090f3058b6ecb9a5447c70ee97 not found: ID does not exist"
Nov 02 12:49:34.085369 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:34.085332    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"]
Nov 02 12:49:34.135875 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:34.135843    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-59878tqbk"]
Nov 02 12:49:34.479617 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:34.479584    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:49:35.581183 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:35.581154    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1463f044-a7fd-490d-a616-2bcc2c540305" path="/var/lib/kubelet/pods/1463f044-a7fd-490d-a616-2bcc2c540305/volumes"
Nov 02 12:49:46.902053 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:49:46.902005    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:49:58.292041 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:49:58.292001    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:50:28.292306 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:50:28.292272    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:50:58.291556 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:50:58.291470    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:51:12.021955 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:12.021925    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:51:12.038599 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:12.038576    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:51:12.059411 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:12.059388    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:51:12.077947 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:12.077924    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:51:22.051886 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:22.051847    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"]
Nov 02 12:51:22.102492 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:22.102459    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7acb3ce-jwjng"]
Nov 02 12:51:23.581064 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:23.581036    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1de5a10-9d79-4eda-b4c2-36f2f923bc96" path="/var/lib/kubelet/pods/f1de5a10-9d79-4eda-b4c2-36f2f923bc96/volumes"
Nov 02 12:51:28.291180 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:51:28.291140    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:51:50.202454 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:50.202422    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-extmod-a-6ac03f_payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz_1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0/payload-processing/0.log"
Nov 02 12:51:55.726366 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.726331    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"]
Nov 02 12:51:55.726824 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.726809    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f7cb7aad-d583-40c7-ab6e-7ce37c712cb6" containerName="payload-processing"
Nov 02 12:51:55.726885 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.726824    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7cb7aad-d583-40c7-ab6e-7ce37c712cb6" containerName="payload-processing"
Nov 02 12:51:55.726885 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.726854    4464 cpu_manager.go:414] "containerMap: removing container" podUID="1463f044-a7fd-490d-a616-2bcc2c540305" containerName="payload-processing"
Nov 02 12:51:55.726885 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.726860    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1463f044-a7fd-490d-a616-2bcc2c540305" containerName="payload-processing"
Nov 02 12:51:55.727067 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.726945    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="1463f044-a7fd-490d-a616-2bcc2c540305" containerName="payload-processing"
Nov 02 12:51:55.727067 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.726961    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7cb7aad-d583-40c7-ab6e-7ce37c712cb6" containerName="payload-processing"
Nov 02 12:51:55.730055 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.730039    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:51:55.781327 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.781297    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49081fd3-bad5-43ac-b28d-1b9604a60bc1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv\" (UID: \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:51:55.781327 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.781330    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49081fd3-bad5-43ac-b28d-1b9604a60bc1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv\" (UID: \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:51:55.781501 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.781442    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49081fd3-bad5-43ac-b28d-1b9604a60bc1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv\" (UID: \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:51:55.781501 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.781473    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft9zd\" (UniqueName: \"kubernetes.io/projected/49081fd3-bad5-43ac-b28d-1b9604a60bc1-kube-api-access-ft9zd\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv\" (UID: \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:51:55.783496 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.783480    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"agc-cred-1ded89c56190\""
Nov 02 12:51:55.798231 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.798205    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"]
Nov 02 12:51:55.882336 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.882302    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49081fd3-bad5-43ac-b28d-1b9604a60bc1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv\" (UID: \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:51:55.882336 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.882338    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49081fd3-bad5-43ac-b28d-1b9604a60bc1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv\" (UID: \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:51:55.882591 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.882404    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49081fd3-bad5-43ac-b28d-1b9604a60bc1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv\" (UID: \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:51:55.882591 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.882429    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft9zd\" (UniqueName: \"kubernetes.io/projected/49081fd3-bad5-43ac-b28d-1b9604a60bc1-kube-api-access-ft9zd\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv\" (UID: \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:51:55.882957 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.882935    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49081fd3-bad5-43ac-b28d-1b9604a60bc1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv\" (UID: \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:51:55.883153 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.883134    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49081fd3-bad5-43ac-b28d-1b9604a60bc1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv\" (UID: \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:51:55.885023 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.885001    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49081fd3-bad5-43ac-b28d-1b9604a60bc1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv\" (UID: \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:51:55.944478 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:55.944453    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft9zd\" (UniqueName: \"kubernetes.io/projected/49081fd3-bad5-43ac-b28d-1b9604a60bc1-kube-api-access-ft9zd\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv\" (UID: \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:51:56.043109 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:56.043028    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:51:56.097840 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:56.097752    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:51:56.417697 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:56.417462    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"]
Nov 02 12:51:56.420876 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:51:56.420844    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49081fd3_bad5_43ac_b28d_1b9604a60bc1.slice/crio-b0b186d13b681b619e71886d44e29803a668c8ff7f66ec8dd69c33da63b844d0 WatchSource:0}: Error finding container b0b186d13b681b619e71886d44e29803a668c8ff7f66ec8dd69c33da63b844d0: Status 404 returned error can't find the container with id b0b186d13b681b619e71886d44e29803a668c8ff7f66ec8dd69c33da63b844d0
Nov 02 12:51:56.633082 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:56.633045    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv" event={"ID":"49081fd3-bad5-43ac-b28d-1b9604a60bc1","Type":"ContainerStarted","Data":"179203cebe26d69237a8393aec8f020f006fa16401523c049920a1f3ef9b1771"}
Nov 02 12:51:56.633288 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:56.633103    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv" event={"ID":"49081fd3-bad5-43ac-b28d-1b9604a60bc1","Type":"ContainerStarted","Data":"b0b186d13b681b619e71886d44e29803a668c8ff7f66ec8dd69c33da63b844d0"}
Nov 02 12:51:56.633288 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:56.633221    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:51:56.752819 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:51:56.752762    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv" podStartSLOduration=1.752745862 podStartE2EDuration="1.752745862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:51:56.751908911 +0000 UTC m=+3345.315617320" watchObservedRunningTime="2026-11-02 12:51:56.752745862 +0000 UTC m=+3345.316454271"
Nov 02 12:51:58.291682 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:51:58.291648    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:52:07.184915 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:07.184873    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:52:07.641309 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:07.641276    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:52:07.811096 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:07.811055    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"]
Nov 02 12:52:12.811880 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:12.811820    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz" podUID="1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0" containerName="payload-processing" containerID="cri-o://874333aafdb5e979b42d4dede2d3920ae0de43af933e0fcc281669c0d3e0c127" gracePeriod=25
Nov 02 12:52:13.066675 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.066610    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:52:13.247248 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.247216    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-provider-credentials\") pod \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\" (UID: \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\") "
Nov 02 12:52:13.247421 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.247300    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-kube-api-access-5bnj6\" (UniqueName: \"kubernetes.io/projected/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-kube-api-access-5bnj6\") pod \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\" (UID: \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\") "
Nov 02 12:52:13.247421 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.247364    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-routing-overlay-volume\") pod \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\" (UID: \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\") "
Nov 02 12:52:13.247421 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.247409    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-plugins-config-volume\") pod \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\" (UID: \"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0\") "
Nov 02 12:52:13.247743 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.247711    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-routing-overlay-volume" pod "1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0" (UID: "1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:13.247814 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.247781    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-plugins-config-volume" pod "1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0" (UID: "1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:13.249410 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.249388    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-provider-credentials" pod "1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0" (UID: "1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:13.249473 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.249414    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-kube-api-access-5bnj6" pod "1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0" (UID: "1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0"). InnerVolumeSpecName "kube-api-access-5bnj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:13.348430 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.348356    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bnj6\" (UniqueName: \"kubernetes.io/projected/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-kube-api-access-5bnj6\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:52:13.348430 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.348380    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:52:13.348430 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.348391    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:52:13.348430 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.348400    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:52:13.722946 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.722911    4464 generic.go:356] "Generic (PLEG): container finished" podID="1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0" containerID="874333aafdb5e979b42d4dede2d3920ae0de43af933e0fcc281669c0d3e0c127" exitCode=0
Nov 02 12:52:13.723158 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.722966    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"
Nov 02 12:52:13.723158 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.723005    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz" event={"ID":"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0","Type":"ContainerDied","Data":"874333aafdb5e979b42d4dede2d3920ae0de43af933e0fcc281669c0d3e0c127"}
Nov 02 12:52:13.723158 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.723064    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz" event={"ID":"1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0","Type":"ContainerDied","Data":"e3108ebe6fea22061af4f33d96557e6f57349a3e33baabc5f1c03a9ca327bf21"}
Nov 02 12:52:13.723158 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.723088    4464 scope.go:122] "RemoveContainer" containerID="874333aafdb5e979b42d4dede2d3920ae0de43af933e0fcc281669c0d3e0c127"
Nov 02 12:52:13.731598 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.731583    4464 scope.go:122] "RemoveContainer" containerID="874333aafdb5e979b42d4dede2d3920ae0de43af933e0fcc281669c0d3e0c127"
Nov 02 12:52:13.731852 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:52:13.731834    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874333aafdb5e979b42d4dede2d3920ae0de43af933e0fcc281669c0d3e0c127\": container with ID starting with 874333aafdb5e979b42d4dede2d3920ae0de43af933e0fcc281669c0d3e0c127 not found: ID does not exist" containerID="874333aafdb5e979b42d4dede2d3920ae0de43af933e0fcc281669c0d3e0c127"
Nov 02 12:52:13.731909 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.731858    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874333aafdb5e979b42d4dede2d3920ae0de43af933e0fcc281669c0d3e0c127"} err="failed to get container status \"874333aafdb5e979b42d4dede2d3920ae0de43af933e0fcc281669c0d3e0c127\": rpc error: code = NotFound desc = could not find container \"874333aafdb5e979b42d4dede2d3920ae0de43af933e0fcc281669c0d3e0c127\": container with ID starting with 874333aafdb5e979b42d4dede2d3920ae0de43af933e0fcc281669c0d3e0c127 not found: ID does not exist"
Nov 02 12:52:13.890655 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.890610    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"]
Nov 02 12:52:13.941177 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:13.941148    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-846975jhz"]
Nov 02 12:52:15.581488 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:52:15.581457    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0" path="/var/lib/kubelet/pods/1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0/volumes"
Nov 02 12:52:28.291363 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:52:28.291326    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:52:58.291305 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:52:58.291265    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:53:14.127559 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:53:14.127521    4464 scope.go:122] "RemoveContainer" containerID="e338507f26a9204381778f97bbae2e4d67c927a46c996df8bd7360f0686910be"
Nov 02 12:53:28.291595 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:53:28.291554    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:53:58.291987 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:53:58.291897    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:54:28.291667 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:54:28.291622    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:54:58.290847 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:54:58.290807    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:55:22.319366 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:55:22.319338    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-extmod-a-6ac03f_payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv_49081fd3-bad5-43ac-b28d-1b9604a60bc1/payload-processing/0.log"
Nov 02 12:55:28.291795 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:55:28.291714    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:55:58.291357 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:55:58.291318    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:56:12.054667 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:56:12.054642    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:56:12.071968 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:56:12.071944    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:56:12.095371 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:56:12.095350    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 12:56:12.112554 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:56:12.112534    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 12:56:28.295188 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:56:28.295153    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:56:58.290882 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:56:58.290805    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:57:28.291268 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:57:28.291226    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:57:58.291848 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:57:58.291743    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:58:28.292281 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:58:28.292198    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:58:28.799337 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:28.799304    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-extmod-a-6ac03f_payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv_49081fd3-bad5-43ac-b28d-1b9604a60bc1/payload-processing/0.log"
Nov 02 12:58:29.615787 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.615754    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"]
Nov 02 12:58:29.616263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.616208    4464 cpu_manager.go:414] "containerMap: removing container" podUID="1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0" containerName="payload-processing"
Nov 02 12:58:29.616263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.616220    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0" containerName="payload-processing"
Nov 02 12:58:29.616341 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.616297    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e301e8f-5107-47d9-8ffc-0b99b9e5d1b0" containerName="payload-processing"
Nov 02 12:58:29.619512 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.619495    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"
Nov 02 12:58:29.673040 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.673019    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:58:29.673259 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.673241    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:58:29.673313 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.673265    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:58:29.673313 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.673299    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:58:29.725461 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.725433    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"]
Nov 02 12:58:29.737906 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.737880    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klshr\" (UniqueName: \"kubernetes.io/projected/4874f4e9-22d3-40ba-bb62-1eb48865dd05-kube-api-access-klshr\") pod \"maas-api-revoke-sub-05b22116-dnpd7\" (UID: \"4874f4e9-22d3-40ba-bb62-1eb48865dd05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"
Nov 02 12:58:29.738054 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.737919    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4874f4e9-22d3-40ba-bb62-1eb48865dd05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05b22116-dnpd7\" (UID: \"4874f4e9-22d3-40ba-bb62-1eb48865dd05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"
Nov 02 12:58:29.738054 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.737945    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4874f4e9-22d3-40ba-bb62-1eb48865dd05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05b22116-dnpd7\" (UID: \"4874f4e9-22d3-40ba-bb62-1eb48865dd05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"
Nov 02 12:58:29.838486 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.838455    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klshr\" (UniqueName: \"kubernetes.io/projected/4874f4e9-22d3-40ba-bb62-1eb48865dd05-kube-api-access-klshr\") pod \"maas-api-revoke-sub-05b22116-dnpd7\" (UID: \"4874f4e9-22d3-40ba-bb62-1eb48865dd05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"
Nov 02 12:58:29.838664 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.838497    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4874f4e9-22d3-40ba-bb62-1eb48865dd05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05b22116-dnpd7\" (UID: \"4874f4e9-22d3-40ba-bb62-1eb48865dd05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"
Nov 02 12:58:29.838664 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.838524    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4874f4e9-22d3-40ba-bb62-1eb48865dd05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05b22116-dnpd7\" (UID: \"4874f4e9-22d3-40ba-bb62-1eb48865dd05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"
Nov 02 12:58:29.839168 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.839145    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4874f4e9-22d3-40ba-bb62-1eb48865dd05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05b22116-dnpd7\" (UID: \"4874f4e9-22d3-40ba-bb62-1eb48865dd05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"
Nov 02 12:58:29.840813 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.840788    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4874f4e9-22d3-40ba-bb62-1eb48865dd05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05b22116-dnpd7\" (UID: \"4874f4e9-22d3-40ba-bb62-1eb48865dd05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"
Nov 02 12:58:29.900552 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.900499    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klshr\" (UniqueName: \"kubernetes.io/projected/4874f4e9-22d3-40ba-bb62-1eb48865dd05-kube-api-access-klshr\") pod \"maas-api-revoke-sub-05b22116-dnpd7\" (UID: \"4874f4e9-22d3-40ba-bb62-1eb48865dd05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"
Nov 02 12:58:29.932534 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:29.932512    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"
Nov 02 12:58:30.303855 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:30.303827    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"]
Nov 02 12:58:30.305650 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:58:30.305622    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4874f4e9_22d3_40ba_bb62_1eb48865dd05.slice/crio-2437acea907de27333163f9fa9c24d25b52594ac272aebe7bc439c97428cd1a7 WatchSource:0}: Error finding container 2437acea907de27333163f9fa9c24d25b52594ac272aebe7bc439c97428cd1a7: Status 404 returned error can't find the container with id 2437acea907de27333163f9fa9c24d25b52594ac272aebe7bc439c97428cd1a7
Nov 02 12:58:30.543733 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:30.543694    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7" event={"ID":"4874f4e9-22d3-40ba-bb62-1eb48865dd05","Type":"ContainerStarted","Data":"2437acea907de27333163f9fa9c24d25b52594ac272aebe7bc439c97428cd1a7"}
Nov 02 12:58:31.549795 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:31.549764    4464 generic.go:356] "Generic (PLEG): container finished" podID="4874f4e9-22d3-40ba-bb62-1eb48865dd05" containerID="ffa0e1b0799f88a469ed5b68977495acab1daf991525974c1309388d3e1d9c9b" exitCode=0
Nov 02 12:58:31.550255 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:31.549810    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7" event={"ID":"4874f4e9-22d3-40ba-bb62-1eb48865dd05","Type":"ContainerDied","Data":"ffa0e1b0799f88a469ed5b68977495acab1daf991525974c1309388d3e1d9c9b"}
Nov 02 12:58:32.695512 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:32.695487    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"
Nov 02 12:58:32.865440 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:32.865369    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4874f4e9-22d3-40ba-bb62-1eb48865dd05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4874f4e9-22d3-40ba-bb62-1eb48865dd05-maas-api-openshift-ca\") pod \"4874f4e9-22d3-40ba-bb62-1eb48865dd05\" (UID: \"4874f4e9-22d3-40ba-bb62-1eb48865dd05\") "
Nov 02 12:58:32.865440 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:32.865420    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4874f4e9-22d3-40ba-bb62-1eb48865dd05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4874f4e9-22d3-40ba-bb62-1eb48865dd05-maas-api-xks-ca\") pod \"4874f4e9-22d3-40ba-bb62-1eb48865dd05\" (UID: \"4874f4e9-22d3-40ba-bb62-1eb48865dd05\") "
Nov 02 12:58:32.865630 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:32.865535    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4874f4e9-22d3-40ba-bb62-1eb48865dd05-kube-api-access-klshr\" (UniqueName: \"kubernetes.io/projected/4874f4e9-22d3-40ba-bb62-1eb48865dd05-kube-api-access-klshr\") pod \"4874f4e9-22d3-40ba-bb62-1eb48865dd05\" (UID: \"4874f4e9-22d3-40ba-bb62-1eb48865dd05\") "
Nov 02 12:58:32.865761 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:32.865736    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4874f4e9-22d3-40ba-bb62-1eb48865dd05-maas-api-openshift-ca" pod "4874f4e9-22d3-40ba-bb62-1eb48865dd05" (UID: "4874f4e9-22d3-40ba-bb62-1eb48865dd05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:58:32.867448 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:32.867420    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4874f4e9-22d3-40ba-bb62-1eb48865dd05-maas-api-xks-ca" pod "4874f4e9-22d3-40ba-bb62-1eb48865dd05" (UID: "4874f4e9-22d3-40ba-bb62-1eb48865dd05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:58:32.867563 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:32.867470    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4874f4e9-22d3-40ba-bb62-1eb48865dd05-kube-api-access-klshr" pod "4874f4e9-22d3-40ba-bb62-1eb48865dd05" (UID: "4874f4e9-22d3-40ba-bb62-1eb48865dd05"). InnerVolumeSpecName "kube-api-access-klshr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:32.966735 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:32.966709    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klshr\" (UniqueName: \"kubernetes.io/projected/4874f4e9-22d3-40ba-bb62-1eb48865dd05-kube-api-access-klshr\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:58:32.966735 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:32.966731    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4874f4e9-22d3-40ba-bb62-1eb48865dd05-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:58:32.966901 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:32.966740    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4874f4e9-22d3-40ba-bb62-1eb48865dd05-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:58:33.560821 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:33.560782    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7" event={"ID":"4874f4e9-22d3-40ba-bb62-1eb48865dd05","Type":"ContainerDied","Data":"2437acea907de27333163f9fa9c24d25b52594ac272aebe7bc439c97428cd1a7"}
Nov 02 12:58:33.560821 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:33.560808    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"
Nov 02 12:58:33.561093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:33.560866    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2437acea907de27333163f9fa9c24d25b52594ac272aebe7bc439c97428cd1a7"
Nov 02 12:58:42.943487 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:42.943453    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"]
Nov 02 12:58:42.944029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:42.944006    4464 cpu_manager.go:414] "containerMap: removing container" podUID="4874f4e9-22d3-40ba-bb62-1eb48865dd05" containerName="revoke-keys"
Nov 02 12:58:42.944029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:42.944023    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4874f4e9-22d3-40ba-bb62-1eb48865dd05" containerName="revoke-keys"
Nov 02 12:58:42.944149 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:42.944135    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="4874f4e9-22d3-40ba-bb62-1eb48865dd05" containerName="revoke-keys"
Nov 02 12:58:42.947408 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:42.947393    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"
Nov 02 12:58:42.971062 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:42.971034    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8257f59d-gm9kj\" (UID: \"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"
Nov 02 12:58:42.971180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:42.971073    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8257f59d-gm9kj\" (UID: \"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"
Nov 02 12:58:42.971180 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:42.971091    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pjnf\" (UniqueName: \"kubernetes.io/projected/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-kube-api-access-4pjnf\") pod \"maas-api-revoke-sub-8257f59d-gm9kj\" (UID: \"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"
Nov 02 12:58:43.000446 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.000421    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:58:43.000589 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.000571    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:58:43.000672 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.000652    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:58:43.000753 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.000736    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:58:43.053349 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.053314    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"]
Nov 02 12:58:43.072240 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.072205    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8257f59d-gm9kj\" (UID: \"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"
Nov 02 12:58:43.072410 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.072253    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8257f59d-gm9kj\" (UID: \"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"
Nov 02 12:58:43.072410 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.072274    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pjnf\" (UniqueName: \"kubernetes.io/projected/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-kube-api-access-4pjnf\") pod \"maas-api-revoke-sub-8257f59d-gm9kj\" (UID: \"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"
Nov 02 12:58:43.072900 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.072872    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8257f59d-gm9kj\" (UID: \"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"
Nov 02 12:58:43.074595 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.074571    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8257f59d-gm9kj\" (UID: \"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"
Nov 02 12:58:43.134203 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.134168    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pjnf\" (UniqueName: \"kubernetes.io/projected/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-kube-api-access-4pjnf\") pod \"maas-api-revoke-sub-8257f59d-gm9kj\" (UID: \"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"
Nov 02 12:58:43.259946 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.259908    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"
Nov 02 12:58:43.317444 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.316858    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"]
Nov 02 12:58:43.324042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.323578    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"
Nov 02 12:58:43.375645 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.375609    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8zjm\" (UniqueName: \"kubernetes.io/projected/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-kube-api-access-w8zjm\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj\" (UID: \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"
Nov 02 12:58:43.375853 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.375666    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj\" (UID: \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"
Nov 02 12:58:43.376003 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.375940    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj\" (UID: \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"
Nov 02 12:58:43.376155 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.376051    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj\" (UID: \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"
Nov 02 12:58:43.378097 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.378073    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"]
Nov 02 12:58:43.477171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.477131    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zjm\" (UniqueName: \"kubernetes.io/projected/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-kube-api-access-w8zjm\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj\" (UID: \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"
Nov 02 12:58:43.477171 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.477176    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj\" (UID: \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"
Nov 02 12:58:43.477431 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.477246    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj\" (UID: \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"
Nov 02 12:58:43.477431 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.477270    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj\" (UID: \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"
Nov 02 12:58:43.478035 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.478006    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj\" (UID: \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"
Nov 02 12:58:43.478165 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.478127    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj\" (UID: \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"
Nov 02 12:58:43.480004 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.479967    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj\" (UID: \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"
Nov 02 12:58:43.539075 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.539003    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8zjm\" (UniqueName: \"kubernetes.io/projected/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-kube-api-access-w8zjm\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj\" (UID: \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"
Nov 02 12:58:43.635694 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.635663    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"]
Nov 02 12:58:43.637201 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:58:43.637172    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a357fb_c9b7_412e_8b29_3ba6cb4718f9.slice/crio-b738259fd63fbfa924e5d09eb73830631cb8359e14b4ad602f8a90d571aa5b34 WatchSource:0}: Error finding container b738259fd63fbfa924e5d09eb73830631cb8359e14b4ad602f8a90d571aa5b34: Status 404 returned error can't find the container with id b738259fd63fbfa924e5d09eb73830631cb8359e14b4ad602f8a90d571aa5b34
Nov 02 12:58:43.639390 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:43.639365    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"
Nov 02 12:58:44.016727 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:44.016700    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"]
Nov 02 12:58:44.018057 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:58:44.018034    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf38a272_fe8a_4cb1_a31b_2a37ca1f906d.slice/crio-be41a4d1473bb6e6d995389bdb4525854745aa013284ce4bc1bfb8cec185fa83 WatchSource:0}: Error finding container be41a4d1473bb6e6d995389bdb4525854745aa013284ce4bc1bfb8cec185fa83: Status 404 returned error can't find the container with id be41a4d1473bb6e6d995389bdb4525854745aa013284ce4bc1bfb8cec185fa83
Nov 02 12:58:44.615874 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:44.615840    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj" event={"ID":"df38a272-fe8a-4cb1-a31b-2a37ca1f906d","Type":"ContainerStarted","Data":"6f75341f8b007164ea774273ba9bd5ea774412792fd880daadbd9c490101e678"}
Nov 02 12:58:44.616064 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:44.615889    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj" event={"ID":"df38a272-fe8a-4cb1-a31b-2a37ca1f906d","Type":"ContainerStarted","Data":"be41a4d1473bb6e6d995389bdb4525854745aa013284ce4bc1bfb8cec185fa83"}
Nov 02 12:58:44.616064 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:44.615953    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"
Nov 02 12:58:44.617552 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:44.617533    4464 generic.go:356] "Generic (PLEG): container finished" podID="d8a357fb-c9b7-412e-8b29-3ba6cb4718f9" containerID="ae8b6a0ce0653b038c91e158abdf370e868aa9b72cd6cf2b3cf91e04bcdf3e5a" exitCode=0
Nov 02 12:58:44.617636 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:44.617616    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj" event={"ID":"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9","Type":"ContainerDied","Data":"ae8b6a0ce0653b038c91e158abdf370e868aa9b72cd6cf2b3cf91e04bcdf3e5a"}
Nov 02 12:58:44.617690 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:44.617656    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj" event={"ID":"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9","Type":"ContainerStarted","Data":"b738259fd63fbfa924e5d09eb73830631cb8359e14b4ad602f8a90d571aa5b34"}
Nov 02 12:58:44.735765 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:44.735722    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj" podStartSLOduration=1.735709547 podStartE2EDuration="1.735709547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:58:44.733940827 +0000 UTC m=+3753.297649262" watchObservedRunningTime="2026-11-02 12:58:44.735709547 +0000 UTC m=+3753.299417956"
Nov 02 12:58:44.879474 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:44.879442    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:58:45.757668 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:45.757646    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"
Nov 02 12:58:45.799429 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:45.799402    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-kube-api-access-4pjnf\" (UniqueName: \"kubernetes.io/projected/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-kube-api-access-4pjnf\") pod \"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9\" (UID: \"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9\") "
Nov 02 12:58:45.799594 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:45.799463    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-maas-api-xks-ca\") pod \"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9\" (UID: \"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9\") "
Nov 02 12:58:45.799594 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:45.799563    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-maas-api-openshift-ca\") pod \"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9\" (UID: \"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9\") "
Nov 02 12:58:45.799931 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:45.799908    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-maas-api-openshift-ca" pod "d8a357fb-c9b7-412e-8b29-3ba6cb4718f9" (UID: "d8a357fb-c9b7-412e-8b29-3ba6cb4718f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:58:45.801486 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:45.801461    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-maas-api-xks-ca" pod "d8a357fb-c9b7-412e-8b29-3ba6cb4718f9" (UID: "d8a357fb-c9b7-412e-8b29-3ba6cb4718f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:58:45.801486 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:45.801470    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-kube-api-access-4pjnf" pod "d8a357fb-c9b7-412e-8b29-3ba6cb4718f9" (UID: "d8a357fb-c9b7-412e-8b29-3ba6cb4718f9"). InnerVolumeSpecName "kube-api-access-4pjnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:45.900363 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:45.900279    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:58:45.900363 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:45.900310    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:58:45.900363 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:45.900324    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pjnf\" (UniqueName: \"kubernetes.io/projected/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9-kube-api-access-4pjnf\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:58:46.628187 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:46.628148    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"
Nov 02 12:58:46.628187 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:46.628172    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj" event={"ID":"d8a357fb-c9b7-412e-8b29-3ba6cb4718f9","Type":"ContainerDied","Data":"b738259fd63fbfa924e5d09eb73830631cb8359e14b4ad602f8a90d571aa5b34"}
Nov 02 12:58:46.628399 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:46.628215    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b738259fd63fbfa924e5d09eb73830631cb8359e14b4ad602f8a90d571aa5b34"
Nov 02 12:58:54.529151 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.529118    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"]
Nov 02 12:58:54.529652 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.529636    4464 cpu_manager.go:414] "containerMap: removing container" podUID="d8a357fb-c9b7-412e-8b29-3ba6cb4718f9" containerName="revoke-keys"
Nov 02 12:58:54.529704 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.529654    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a357fb-c9b7-412e-8b29-3ba6cb4718f9" containerName="revoke-keys"
Nov 02 12:58:54.529745 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.529726    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a357fb-c9b7-412e-8b29-3ba6cb4718f9" containerName="revoke-keys"
Nov 02 12:58:54.532818 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.532801    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"
Nov 02 12:58:54.580037 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.580009    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfksr\" (UniqueName: \"kubernetes.io/projected/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-kube-api-access-lfksr\") pod \"maas-api-revoke-sub-524e6df4-l2jm8\" (UID: \"8d9bbc60-8d04-45d8-a35a-c90392f41bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"
Nov 02 12:58:54.580174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.580066    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-524e6df4-l2jm8\" (UID: \"8d9bbc60-8d04-45d8-a35a-c90392f41bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"
Nov 02 12:58:54.580174 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.580147    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-524e6df4-l2jm8\" (UID: \"8d9bbc60-8d04-45d8-a35a-c90392f41bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"
Nov 02 12:58:54.638060 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.638026    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:58:54.638230 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.638070    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:58:54.638230 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.638149    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:58:54.638365 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.638230    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:58:54.640527 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.640500    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"]
Nov 02 12:58:54.681166 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.681139    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-524e6df4-l2jm8\" (UID: \"8d9bbc60-8d04-45d8-a35a-c90392f41bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"
Nov 02 12:58:54.681337 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.681172    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-524e6df4-l2jm8\" (UID: \"8d9bbc60-8d04-45d8-a35a-c90392f41bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"
Nov 02 12:58:54.681337 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.681250    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfksr\" (UniqueName: \"kubernetes.io/projected/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-kube-api-access-lfksr\") pod \"maas-api-revoke-sub-524e6df4-l2jm8\" (UID: \"8d9bbc60-8d04-45d8-a35a-c90392f41bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"
Nov 02 12:58:54.681863 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.681840    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-524e6df4-l2jm8\" (UID: \"8d9bbc60-8d04-45d8-a35a-c90392f41bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"
Nov 02 12:58:54.683583 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.683560    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-524e6df4-l2jm8\" (UID: \"8d9bbc60-8d04-45d8-a35a-c90392f41bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"
Nov 02 12:58:54.742935 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.742908    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfksr\" (UniqueName: \"kubernetes.io/projected/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-kube-api-access-lfksr\") pod \"maas-api-revoke-sub-524e6df4-l2jm8\" (UID: \"8d9bbc60-8d04-45d8-a35a-c90392f41bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"
Nov 02 12:58:54.846067 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:54.846006    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"
Nov 02 12:58:55.219221 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:55.219194    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"]
Nov 02 12:58:55.220762 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:58:55.220734    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9bbc60_8d04_45d8_a35a_c90392f41bb1.slice/crio-424473d47458be638f8427f6ab0391bbfa49ad224eb08b3267c6df38e3111dd9 WatchSource:0}: Error finding container 424473d47458be638f8427f6ab0391bbfa49ad224eb08b3267c6df38e3111dd9: Status 404 returned error can't find the container with id 424473d47458be638f8427f6ab0391bbfa49ad224eb08b3267c6df38e3111dd9
Nov 02 12:58:55.625553 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:55.625514    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj" podUID="df38a272-fe8a-4cb1-a31b-2a37ca1f906d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:58:55.675160 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:55.675122    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8" event={"ID":"8d9bbc60-8d04-45d8-a35a-c90392f41bb1","Type":"ContainerStarted","Data":"2f910a44ba251989a92e9ca4b373bf45d2ddc7911c3fa2efa2fa53b10d7556d6"}
Nov 02 12:58:55.675330 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:55.675178    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8" event={"ID":"8d9bbc60-8d04-45d8-a35a-c90392f41bb1","Type":"ContainerStarted","Data":"424473d47458be638f8427f6ab0391bbfa49ad224eb08b3267c6df38e3111dd9"}
Nov 02 12:58:55.794456 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:55.794356    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8" podStartSLOduration=1.7943419569999999 podStartE2EDuration="1.794341957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:58:55.792639654 +0000 UTC m=+3764.356348066" watchObservedRunningTime="2026-11-02 12:58:55.794341957 +0000 UTC m=+3764.358050366"
Nov 02 12:58:56.681172 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:56.681137    4464 generic.go:356] "Generic (PLEG): container finished" podID="8d9bbc60-8d04-45d8-a35a-c90392f41bb1" containerID="2f910a44ba251989a92e9ca4b373bf45d2ddc7911c3fa2efa2fa53b10d7556d6" exitCode=0
Nov 02 12:58:56.681556 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:56.681224    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8" event={"ID":"8d9bbc60-8d04-45d8-a35a-c90392f41bb1","Type":"ContainerDied","Data":"2f910a44ba251989a92e9ca4b373bf45d2ddc7911c3fa2efa2fa53b10d7556d6"}
Nov 02 12:58:57.777228 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:57.777185    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:58:57.833815 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:57.833785    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"
Nov 02 12:58:57.912198 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:57.912163    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-kube-api-access-lfksr\" (UniqueName: \"kubernetes.io/projected/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-kube-api-access-lfksr\") pod \"8d9bbc60-8d04-45d8-a35a-c90392f41bb1\" (UID: \"8d9bbc60-8d04-45d8-a35a-c90392f41bb1\") "
Nov 02 12:58:57.912357 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:57.912292    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-maas-api-openshift-ca\") pod \"8d9bbc60-8d04-45d8-a35a-c90392f41bb1\" (UID: \"8d9bbc60-8d04-45d8-a35a-c90392f41bb1\") "
Nov 02 12:58:57.912436 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:57.912384    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-maas-api-xks-ca\") pod \"8d9bbc60-8d04-45d8-a35a-c90392f41bb1\" (UID: \"8d9bbc60-8d04-45d8-a35a-c90392f41bb1\") "
Nov 02 12:58:57.913118 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:57.912706    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-maas-api-openshift-ca" pod "8d9bbc60-8d04-45d8-a35a-c90392f41bb1" (UID: "8d9bbc60-8d04-45d8-a35a-c90392f41bb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:58:57.914958 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:57.914876    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-maas-api-xks-ca" pod "8d9bbc60-8d04-45d8-a35a-c90392f41bb1" (UID: "8d9bbc60-8d04-45d8-a35a-c90392f41bb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:58:57.915107 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:57.915037    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-kube-api-access-lfksr" pod "8d9bbc60-8d04-45d8-a35a-c90392f41bb1" (UID: "8d9bbc60-8d04-45d8-a35a-c90392f41bb1"). InnerVolumeSpecName "kube-api-access-lfksr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:58.013998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:58.013941    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:58:58.013998 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:58.013969    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:58:58.014188 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:58.014005    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfksr\" (UniqueName: \"kubernetes.io/projected/8d9bbc60-8d04-45d8-a35a-c90392f41bb1-kube-api-access-lfksr\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:58:58.291162 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:58:58.291075    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:58:58.691934 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:58.691855    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"
Nov 02 12:58:58.691934 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:58.691879    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8" event={"ID":"8d9bbc60-8d04-45d8-a35a-c90392f41bb1","Type":"ContainerDied","Data":"424473d47458be638f8427f6ab0391bbfa49ad224eb08b3267c6df38e3111dd9"}
Nov 02 12:58:58.691934 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:58:58.691924    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="424473d47458be638f8427f6ab0391bbfa49ad224eb08b3267c6df38e3111dd9"
Nov 02 12:59:05.623894 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:05.623847    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj" podUID="df38a272-fe8a-4cb1-a31b-2a37ca1f906d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:59:07.991294 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:07.991256    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"]
Nov 02 12:59:07.992640 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:07.992607    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj" podUID="df38a272-fe8a-4cb1-a31b-2a37ca1f906d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:59:08.041339 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.041303    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"]
Nov 02 12:59:08.041771 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.041759    4464 cpu_manager.go:414] "containerMap: removing container" podUID="8d9bbc60-8d04-45d8-a35a-c90392f41bb1" containerName="revoke-keys"
Nov 02 12:59:08.041816 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.041773    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d9bbc60-8d04-45d8-a35a-c90392f41bb1" containerName="revoke-keys"
Nov 02 12:59:08.041872 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.041862    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d9bbc60-8d04-45d8-a35a-c90392f41bb1" containerName="revoke-keys"
Nov 02 12:59:08.045312 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.045297    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"
Nov 02 12:59:08.102684 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.102650    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"]
Nov 02 12:59:08.105958 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.105924    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4934733e-ba9a-42a0-8a48-ee258fafea16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74\" (UID: \"4934733e-ba9a-42a0-8a48-ee258fafea16\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"
Nov 02 12:59:08.106149 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.106068    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4934733e-ba9a-42a0-8a48-ee258fafea16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74\" (UID: \"4934733e-ba9a-42a0-8a48-ee258fafea16\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"
Nov 02 12:59:08.106226 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.106148    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4qgg\" (UniqueName: \"kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-kube-api-access-m4qgg\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74\" (UID: \"4934733e-ba9a-42a0-8a48-ee258fafea16\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"
Nov 02 12:59:08.106307 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.106281    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74\" (UID: \"4934733e-ba9a-42a0-8a48-ee258fafea16\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"
Nov 02 12:59:08.207557 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.207520    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4qgg\" (UniqueName: \"kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-kube-api-access-m4qgg\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74\" (UID: \"4934733e-ba9a-42a0-8a48-ee258fafea16\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"
Nov 02 12:59:08.207739 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.207565    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74\" (UID: \"4934733e-ba9a-42a0-8a48-ee258fafea16\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"
Nov 02 12:59:08.207739 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.207613    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4934733e-ba9a-42a0-8a48-ee258fafea16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74\" (UID: \"4934733e-ba9a-42a0-8a48-ee258fafea16\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"
Nov 02 12:59:08.207739 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.207648    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4934733e-ba9a-42a0-8a48-ee258fafea16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74\" (UID: \"4934733e-ba9a-42a0-8a48-ee258fafea16\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"
Nov 02 12:59:08.208288 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.208262    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4934733e-ba9a-42a0-8a48-ee258fafea16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74\" (UID: \"4934733e-ba9a-42a0-8a48-ee258fafea16\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"
Nov 02 12:59:08.208498 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.208477    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4934733e-ba9a-42a0-8a48-ee258fafea16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74\" (UID: \"4934733e-ba9a-42a0-8a48-ee258fafea16\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"
Nov 02 12:59:08.210086 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.210065    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74\" (UID: \"4934733e-ba9a-42a0-8a48-ee258fafea16\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"
Nov 02 12:59:08.269433 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.269360    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4qgg\" (UniqueName: \"kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-kube-api-access-m4qgg\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74\" (UID: \"4934733e-ba9a-42a0-8a48-ee258fafea16\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"
Nov 02 12:59:08.358177 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.358152    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"
Nov 02 12:59:08.732953 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.732924    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"]
Nov 02 12:59:08.734394 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:59:08.734368    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4934733e_ba9a_42a0_8a48_ee258fafea16.slice/crio-60ba33dacd57bf0978a85bf392dc430efbea31d34badf329518d4c9e5b35d23c WatchSource:0}: Error finding container 60ba33dacd57bf0978a85bf392dc430efbea31d34badf329518d4c9e5b35d23c: Status 404 returned error can't find the container with id 60ba33dacd57bf0978a85bf392dc430efbea31d34badf329518d4c9e5b35d23c
Nov 02 12:59:08.746364 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:08.746337    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74" event={"ID":"4934733e-ba9a-42a0-8a48-ee258fafea16","Type":"ContainerStarted","Data":"60ba33dacd57bf0978a85bf392dc430efbea31d34badf329518d4c9e5b35d23c"}
Nov 02 12:59:09.753083 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:09.753043    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74" event={"ID":"4934733e-ba9a-42a0-8a48-ee258fafea16","Type":"ContainerStarted","Data":"f0ba85d260ae029a79e4a49563de14c31b434cec5f895de617ac35cbe1211279"}
Nov 02 12:59:09.753500 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:09.753115    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"
Nov 02 12:59:09.825352 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:09.825321    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-c501215b3849: secret "agc-cred-c501215b3849" not found
Nov 02 12:59:09.825352 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:09.825357    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74: secret "agc-cred-c501215b3849" not found
Nov 02 12:59:09.825606 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:09.825439    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-provider-credentials podName:4934733e-ba9a-42a0-8a48-ee258fafea16 nodeName:}" failed. No retries permitted until 2026-11-02 12:59:10.325418884 +0000 UTC m=+3778.889127292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74" (UID: "4934733e-ba9a-42a0-8a48-ee258fafea16") : secret "agc-cred-c501215b3849" not found
Nov 02 12:59:09.873309 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:09.873260    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74" podStartSLOduration=2.873246741 podStartE2EDuration="2.873246741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:59:09.871034405 +0000 UTC m=+3778.434742819" watchObservedRunningTime="2026-11-02 12:59:09.873246741 +0000 UTC m=+3778.436955180"
Nov 02 12:59:10.040257 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.040179    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"]
Nov 02 12:59:10.044017 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.043995    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"
Nov 02 12:59:10.098623 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.098598    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:59:10.098623 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.098599    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:59:10.098864 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.098603    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 12:59:10.098864 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.098624    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:59:10.101274 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.101250    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"]
Nov 02 12:59:10.128249 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.128215    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m424r\" (UniqueName: \"kubernetes.io/projected/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-kube-api-access-m424r\") pod \"maas-api-revoke-sub-edcd0990-n7b7m\" (UID: \"40e32b65-70ea-42cd-aa32-dbd4bd704fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"
Nov 02 12:59:10.128349 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.128283    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edcd0990-n7b7m\" (UID: \"40e32b65-70ea-42cd-aa32-dbd4bd704fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"
Nov 02 12:59:10.128398 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.128371    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edcd0990-n7b7m\" (UID: \"40e32b65-70ea-42cd-aa32-dbd4bd704fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"
Nov 02 12:59:10.229964 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.229930    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m424r\" (UniqueName: \"kubernetes.io/projected/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-kube-api-access-m424r\") pod \"maas-api-revoke-sub-edcd0990-n7b7m\" (UID: \"40e32b65-70ea-42cd-aa32-dbd4bd704fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"
Nov 02 12:59:10.230159 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.230001    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edcd0990-n7b7m\" (UID: \"40e32b65-70ea-42cd-aa32-dbd4bd704fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"
Nov 02 12:59:10.230159 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.230026    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edcd0990-n7b7m\" (UID: \"40e32b65-70ea-42cd-aa32-dbd4bd704fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"
Nov 02 12:59:10.230620 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.230600    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edcd0990-n7b7m\" (UID: \"40e32b65-70ea-42cd-aa32-dbd4bd704fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"
Nov 02 12:59:10.232317 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.232293    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edcd0990-n7b7m\" (UID: \"40e32b65-70ea-42cd-aa32-dbd4bd704fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"
Nov 02 12:59:10.291907 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.291838    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m424r\" (UniqueName: \"kubernetes.io/projected/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-kube-api-access-m424r\") pod \"maas-api-revoke-sub-edcd0990-n7b7m\" (UID: \"40e32b65-70ea-42cd-aa32-dbd4bd704fb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"
Nov 02 12:59:10.330839 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:10.330813    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-c501215b3849: secret "agc-cred-c501215b3849" not found
Nov 02 12:59:10.330839 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:10.330843    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74: secret "agc-cred-c501215b3849" not found
Nov 02 12:59:10.331030 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:10.330892    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-provider-credentials podName:4934733e-ba9a-42a0-8a48-ee258fafea16 nodeName:}" failed. No retries permitted until 2026-11-02 12:59:11.330878516 +0000 UTC m=+3779.894586920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74" (UID: "4934733e-ba9a-42a0-8a48-ee258fafea16") : secret "agc-cred-c501215b3849" not found
Nov 02 12:59:10.357783 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.357760    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"
Nov 02 12:59:10.693600 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.693527    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"]
Nov 02 12:59:10.725598 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.725566    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"]
Nov 02 12:59:10.730524 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.730503    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"]
Nov 02 12:59:10.730655 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.730594    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"
Nov 02 12:59:10.767114 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:59:10.767089    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e32b65_70ea_42cd_aa32_dbd4bd704fb9.slice/crio-f7e91d288807a275776e38bc5189de6192237dcf5524579cb267e0626bb9fabe WatchSource:0}: Error finding container f7e91d288807a275776e38bc5189de6192237dcf5524579cb267e0626bb9fabe: Status 404 returned error can't find the container with id f7e91d288807a275776e38bc5189de6192237dcf5524579cb267e0626bb9fabe
Nov 02 12:59:10.809211 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.807819    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"]
Nov 02 12:59:10.836895 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.836866    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bad023-80b3-4fad-9359-44639b61a21a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc\" (UID: \"79bad023-80b3-4fad-9359-44639b61a21a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"
Nov 02 12:59:10.837062 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.836916    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8xk8\" (UniqueName: \"kubernetes.io/projected/79bad023-80b3-4fad-9359-44639b61a21a-kube-api-access-b8xk8\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc\" (UID: \"79bad023-80b3-4fad-9359-44639b61a21a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"
Nov 02 12:59:10.837204 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.837181    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79bad023-80b3-4fad-9359-44639b61a21a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc\" (UID: \"79bad023-80b3-4fad-9359-44639b61a21a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"
Nov 02 12:59:10.837287 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.837275    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79bad023-80b3-4fad-9359-44639b61a21a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc\" (UID: \"79bad023-80b3-4fad-9359-44639b61a21a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"
Nov 02 12:59:10.938259 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.938229    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bad023-80b3-4fad-9359-44639b61a21a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc\" (UID: \"79bad023-80b3-4fad-9359-44639b61a21a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"
Nov 02 12:59:10.938429 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.938267    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8xk8\" (UniqueName: \"kubernetes.io/projected/79bad023-80b3-4fad-9359-44639b61a21a-kube-api-access-b8xk8\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc\" (UID: \"79bad023-80b3-4fad-9359-44639b61a21a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"
Nov 02 12:59:10.938429 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.938290    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79bad023-80b3-4fad-9359-44639b61a21a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc\" (UID: \"79bad023-80b3-4fad-9359-44639b61a21a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"
Nov 02 12:59:10.938429 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.938329    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79bad023-80b3-4fad-9359-44639b61a21a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc\" (UID: \"79bad023-80b3-4fad-9359-44639b61a21a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"
Nov 02 12:59:10.938968 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.938937    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bad023-80b3-4fad-9359-44639b61a21a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc\" (UID: \"79bad023-80b3-4fad-9359-44639b61a21a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"
Nov 02 12:59:10.939120 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.939034    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79bad023-80b3-4fad-9359-44639b61a21a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc\" (UID: \"79bad023-80b3-4fad-9359-44639b61a21a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"
Nov 02 12:59:10.940818 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:10.940795    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79bad023-80b3-4fad-9359-44639b61a21a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc\" (UID: \"79bad023-80b3-4fad-9359-44639b61a21a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"
Nov 02 12:59:11.000523 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:11.000497    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8xk8\" (UniqueName: \"kubernetes.io/projected/79bad023-80b3-4fad-9359-44639b61a21a-kube-api-access-b8xk8\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc\" (UID: \"79bad023-80b3-4fad-9359-44639b61a21a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"
Nov 02 12:59:11.043766 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:11.043737    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"
Nov 02 12:59:11.343675 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:11.343596    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-c501215b3849: secret "agc-cred-c501215b3849" not found
Nov 02 12:59:11.343675 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:11.343629    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74: secret "agc-cred-c501215b3849" not found
Nov 02 12:59:11.343859 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:11.343678    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-provider-credentials podName:4934733e-ba9a-42a0-8a48-ee258fafea16 nodeName:}" failed. No retries permitted until 2026-11-02 12:59:13.34366535 +0000 UTC m=+3781.907373737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74" (UID: "4934733e-ba9a-42a0-8a48-ee258fafea16") : secret "agc-cred-c501215b3849" not found
Nov 02 12:59:11.419588 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:11.419566    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"]
Nov 02 12:59:11.420883 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:59:11.420858    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bad023_80b3_4fad_9359_44639b61a21a.slice/crio-0bf9d40236657611e37d469e77944cf7bac55ee835d247b108c6bc352a967b94 WatchSource:0}: Error finding container 0bf9d40236657611e37d469e77944cf7bac55ee835d247b108c6bc352a967b94: Status 404 returned error can't find the container with id 0bf9d40236657611e37d469e77944cf7bac55ee835d247b108c6bc352a967b94
Nov 02 12:59:11.764165 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:11.764134    4464 generic.go:356] "Generic (PLEG): container finished" podID="40e32b65-70ea-42cd-aa32-dbd4bd704fb9" containerID="f5a033c18ae532cf891832d54711982c49a61bef8967abfdf1e580677a238d49" exitCode=0
Nov 02 12:59:11.764330 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:11.764214    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m" event={"ID":"40e32b65-70ea-42cd-aa32-dbd4bd704fb9","Type":"ContainerDied","Data":"f5a033c18ae532cf891832d54711982c49a61bef8967abfdf1e580677a238d49"}
Nov 02 12:59:11.764330 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:11.764265    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m" event={"ID":"40e32b65-70ea-42cd-aa32-dbd4bd704fb9","Type":"ContainerStarted","Data":"f7e91d288807a275776e38bc5189de6192237dcf5524579cb267e0626bb9fabe"}
Nov 02 12:59:11.765791 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:11.765769    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc" event={"ID":"79bad023-80b3-4fad-9359-44639b61a21a","Type":"ContainerStarted","Data":"2b6ca6e8c7c4442ab81c0462cc3ec254351d96f8943b63f1512ca4d2bbc099a1"}
Nov 02 12:59:11.765919 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:11.765886    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"
Nov 02 12:59:11.766000 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:11.765918    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc" event={"ID":"79bad023-80b3-4fad-9359-44639b61a21a","Type":"ContainerStarted","Data":"0bf9d40236657611e37d469e77944cf7bac55ee835d247b108c6bc352a967b94"}
Nov 02 12:59:12.004074 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:12.004030    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc" podStartSLOduration=2.004015593 podStartE2EDuration="2.004015593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:59:12.0022614 +0000 UTC m=+3780.565969809" watchObservedRunningTime="2026-11-02 12:59:12.004015593 +0000 UTC m=+3780.567724001"
Nov 02 12:59:12.098151 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:12.098119    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:59:12.908699 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:12.908678    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"
Nov 02 12:59:12.992233 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:12.992200    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj" podUID="df38a272-fe8a-4cb1-a31b-2a37ca1f906d" containerName="payload-processing" containerID="cri-o://6f75341f8b007164ea774273ba9bd5ea774412792fd880daadbd9c490101e678" gracePeriod=25
Nov 02 12:59:13.063029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.062939    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-maas-api-xks-ca\") pod \"40e32b65-70ea-42cd-aa32-dbd4bd704fb9\" (UID: \"40e32b65-70ea-42cd-aa32-dbd4bd704fb9\") "
Nov 02 12:59:13.063029 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.063022    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-kube-api-access-m424r\" (UniqueName: \"kubernetes.io/projected/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-kube-api-access-m424r\") pod \"40e32b65-70ea-42cd-aa32-dbd4bd704fb9\" (UID: \"40e32b65-70ea-42cd-aa32-dbd4bd704fb9\") "
Nov 02 12:59:13.063487 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.063120    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-maas-api-openshift-ca\") pod \"40e32b65-70ea-42cd-aa32-dbd4bd704fb9\" (UID: \"40e32b65-70ea-42cd-aa32-dbd4bd704fb9\") "
Nov 02 12:59:13.063487 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.063418    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-maas-api-openshift-ca" pod "40e32b65-70ea-42cd-aa32-dbd4bd704fb9" (UID: "40e32b65-70ea-42cd-aa32-dbd4bd704fb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:13.063580 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.063564    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:13.065104 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.065084    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-maas-api-xks-ca" pod "40e32b65-70ea-42cd-aa32-dbd4bd704fb9" (UID: "40e32b65-70ea-42cd-aa32-dbd4bd704fb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:59:13.065179 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.065163    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-kube-api-access-m424r" pod "40e32b65-70ea-42cd-aa32-dbd4bd704fb9" (UID: "40e32b65-70ea-42cd-aa32-dbd4bd704fb9"). InnerVolumeSpecName "kube-api-access-m424r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:13.165023 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.164994    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m424r\" (UniqueName: \"kubernetes.io/projected/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-kube-api-access-m424r\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:13.165159 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.165025    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e32b65-70ea-42cd-aa32-dbd4bd704fb9-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:13.227322 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.227300    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"
Nov 02 12:59:13.366540 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.366468    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-provider-credentials\") pod \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\" (UID: \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\") "
Nov 02 12:59:13.366708 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.366692    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-plugins-config-volume\") pod \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\" (UID: \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\") "
Nov 02 12:59:13.366764 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.366752    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-routing-overlay-volume\") pod \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\" (UID: \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\") "
Nov 02 12:59:13.366860 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.366845    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-kube-api-access-w8zjm\" (UniqueName: \"kubernetes.io/projected/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-kube-api-access-w8zjm\") pod \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\" (UID: \"df38a272-fe8a-4cb1-a31b-2a37ca1f906d\") "
Nov 02 12:59:13.367186 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.367166    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-plugins-config-volume" pod "df38a272-fe8a-4cb1-a31b-2a37ca1f906d" (UID: "df38a272-fe8a-4cb1-a31b-2a37ca1f906d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:13.367309 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.367287    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:13.367368 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:13.367350    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-c501215b3849: secret "agc-cred-c501215b3849" not found
Nov 02 12:59:13.367412 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:13.367385    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74: secret "agc-cred-c501215b3849" not found
Nov 02 12:59:13.367462 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:13.367447    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-provider-credentials podName:4934733e-ba9a-42a0-8a48-ee258fafea16 nodeName:}" failed. No retries permitted until 2026-11-02 12:59:17.367427279 +0000 UTC m=+3785.931135686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74" (UID: "4934733e-ba9a-42a0-8a48-ee258fafea16") : secret "agc-cred-c501215b3849" not found
Nov 02 12:59:13.367693 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.367671    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-routing-overlay-volume" pod "df38a272-fe8a-4cb1-a31b-2a37ca1f906d" (UID: "df38a272-fe8a-4cb1-a31b-2a37ca1f906d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:13.368778 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.368749    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-provider-credentials" pod "df38a272-fe8a-4cb1-a31b-2a37ca1f906d" (UID: "df38a272-fe8a-4cb1-a31b-2a37ca1f906d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:13.369099 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.369077    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-kube-api-access-w8zjm" pod "df38a272-fe8a-4cb1-a31b-2a37ca1f906d" (UID: "df38a272-fe8a-4cb1-a31b-2a37ca1f906d"). InnerVolumeSpecName "kube-api-access-w8zjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:13.468244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.468215    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:13.468244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.468239    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8zjm\" (UniqueName: \"kubernetes.io/projected/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-kube-api-access-w8zjm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:13.468244 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.468248    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df38a272-fe8a-4cb1-a31b-2a37ca1f906d-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:13.777150 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.777121    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"
Nov 02 12:59:13.777305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.777162    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m" event={"ID":"40e32b65-70ea-42cd-aa32-dbd4bd704fb9","Type":"ContainerDied","Data":"f7e91d288807a275776e38bc5189de6192237dcf5524579cb267e0626bb9fabe"}
Nov 02 12:59:13.777305 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.777217    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7e91d288807a275776e38bc5189de6192237dcf5524579cb267e0626bb9fabe"
Nov 02 12:59:13.778621 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.778599    4464 generic.go:356] "Generic (PLEG): container finished" podID="df38a272-fe8a-4cb1-a31b-2a37ca1f906d" containerID="6f75341f8b007164ea774273ba9bd5ea774412792fd880daadbd9c490101e678" exitCode=0
Nov 02 12:59:13.778729 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.778647    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj" event={"ID":"df38a272-fe8a-4cb1-a31b-2a37ca1f906d","Type":"ContainerDied","Data":"6f75341f8b007164ea774273ba9bd5ea774412792fd880daadbd9c490101e678"}
Nov 02 12:59:13.778729 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.778652    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"
Nov 02 12:59:13.778729 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.778685    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj" event={"ID":"df38a272-fe8a-4cb1-a31b-2a37ca1f906d","Type":"ContainerDied","Data":"be41a4d1473bb6e6d995389bdb4525854745aa013284ce4bc1bfb8cec185fa83"}
Nov 02 12:59:13.778729 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.778708    4464 scope.go:122] "RemoveContainer" containerID="6f75341f8b007164ea774273ba9bd5ea774412792fd880daadbd9c490101e678"
Nov 02 12:59:13.790468 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.790345    4464 scope.go:122] "RemoveContainer" containerID="6f75341f8b007164ea774273ba9bd5ea774412792fd880daadbd9c490101e678"
Nov 02 12:59:13.790941 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:13.790915    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f75341f8b007164ea774273ba9bd5ea774412792fd880daadbd9c490101e678\": container with ID starting with 6f75341f8b007164ea774273ba9bd5ea774412792fd880daadbd9c490101e678 not found: ID does not exist" containerID="6f75341f8b007164ea774273ba9bd5ea774412792fd880daadbd9c490101e678"
Nov 02 12:59:13.791055 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.790951    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f75341f8b007164ea774273ba9bd5ea774412792fd880daadbd9c490101e678"} err="failed to get container status \"6f75341f8b007164ea774273ba9bd5ea774412792fd880daadbd9c490101e678\": rpc error: code = NotFound desc = could not find container \"6f75341f8b007164ea774273ba9bd5ea774412792fd880daadbd9c490101e678\": container with ID starting with 6f75341f8b007164ea774273ba9bd5ea774412792fd880daadbd9c490101e678 not found: ID does not exist"
Nov 02 12:59:13.946681 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.946647    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"]
Nov 02 12:59:13.996730 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:13.996702    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-8469w5lxj"]
Nov 02 12:59:15.580450 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:15.580421    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df38a272-fe8a-4cb1-a31b-2a37ca1f906d" path="/var/lib/kubelet/pods/df38a272-fe8a-4cb1-a31b-2a37ca1f906d/volumes"
Nov 02 12:59:16.766490 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:16.766446    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74" podUID="4934733e-ba9a-42a0-8a48-ee258fafea16" containerName="payload-processing" containerID="cri-o://f0ba85d260ae029a79e4a49563de14c31b434cec5f895de617ac35cbe1211279" gracePeriod=25
Nov 02 12:59:17.019535 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.019474    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"
Nov 02 12:59:17.205817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.205786    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4934733e-ba9a-42a0-8a48-ee258fafea16-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4934733e-ba9a-42a0-8a48-ee258fafea16-routing-overlay-volume\") pod \"4934733e-ba9a-42a0-8a48-ee258fafea16\" (UID: \"4934733e-ba9a-42a0-8a48-ee258fafea16\") "
Nov 02 12:59:17.206013 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.205843    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4934733e-ba9a-42a0-8a48-ee258fafea16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4934733e-ba9a-42a0-8a48-ee258fafea16-plugins-config-volume\") pod \"4934733e-ba9a-42a0-8a48-ee258fafea16\" (UID: \"4934733e-ba9a-42a0-8a48-ee258fafea16\") "
Nov 02 12:59:17.206013 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.205892    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-kube-api-access-m4qgg\" (UniqueName: \"kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-kube-api-access-m4qgg\") pod \"4934733e-ba9a-42a0-8a48-ee258fafea16\" (UID: \"4934733e-ba9a-42a0-8a48-ee258fafea16\") "
Nov 02 12:59:17.206163 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.206069    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-provider-credentials\") pod \"4934733e-ba9a-42a0-8a48-ee258fafea16\" (UID: \"4934733e-ba9a-42a0-8a48-ee258fafea16\") "
Nov 02 12:59:17.206281 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.206260    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4934733e-ba9a-42a0-8a48-ee258fafea16-routing-overlay-volume" pod "4934733e-ba9a-42a0-8a48-ee258fafea16" (UID: "4934733e-ba9a-42a0-8a48-ee258fafea16"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:17.206281 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.206266    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4934733e-ba9a-42a0-8a48-ee258fafea16-plugins-config-volume" pod "4934733e-ba9a-42a0-8a48-ee258fafea16" (UID: "4934733e-ba9a-42a0-8a48-ee258fafea16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:17.206425 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.206411    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4934733e-ba9a-42a0-8a48-ee258fafea16-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:17.206469 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.206427    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4934733e-ba9a-42a0-8a48-ee258fafea16-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:17.207963 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.207942    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-provider-credentials" pod "4934733e-ba9a-42a0-8a48-ee258fafea16" (UID: "4934733e-ba9a-42a0-8a48-ee258fafea16"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:17.208050 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.207969    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-kube-api-access-m4qgg" pod "4934733e-ba9a-42a0-8a48-ee258fafea16" (UID: "4934733e-ba9a-42a0-8a48-ee258fafea16"). InnerVolumeSpecName "kube-api-access-m4qgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:17.307002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.306908    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:17.307002 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.306936    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4qgg\" (UniqueName: \"kubernetes.io/projected/4934733e-ba9a-42a0-8a48-ee258fafea16-kube-api-access-m4qgg\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:17.800156 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.800121    4464 generic.go:356] "Generic (PLEG): container finished" podID="4934733e-ba9a-42a0-8a48-ee258fafea16" containerID="f0ba85d260ae029a79e4a49563de14c31b434cec5f895de617ac35cbe1211279" exitCode=0
Nov 02 12:59:17.800590 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.800174    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"
Nov 02 12:59:17.800590 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.800187    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74" event={"ID":"4934733e-ba9a-42a0-8a48-ee258fafea16","Type":"ContainerDied","Data":"f0ba85d260ae029a79e4a49563de14c31b434cec5f895de617ac35cbe1211279"}
Nov 02 12:59:17.800590 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.800237    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74" event={"ID":"4934733e-ba9a-42a0-8a48-ee258fafea16","Type":"ContainerDied","Data":"60ba33dacd57bf0978a85bf392dc430efbea31d34badf329518d4c9e5b35d23c"}
Nov 02 12:59:17.800590 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.800269    4464 scope.go:122] "RemoveContainer" containerID="f0ba85d260ae029a79e4a49563de14c31b434cec5f895de617ac35cbe1211279"
Nov 02 12:59:17.809042 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.809024    4464 scope.go:122] "RemoveContainer" containerID="f0ba85d260ae029a79e4a49563de14c31b434cec5f895de617ac35cbe1211279"
Nov 02 12:59:17.809285 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:17.809266    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ba85d260ae029a79e4a49563de14c31b434cec5f895de617ac35cbe1211279\": container with ID starting with f0ba85d260ae029a79e4a49563de14c31b434cec5f895de617ac35cbe1211279 not found: ID does not exist" containerID="f0ba85d260ae029a79e4a49563de14c31b434cec5f895de617ac35cbe1211279"
Nov 02 12:59:17.809333 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.809293    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ba85d260ae029a79e4a49563de14c31b434cec5f895de617ac35cbe1211279"} err="failed to get container status \"f0ba85d260ae029a79e4a49563de14c31b434cec5f895de617ac35cbe1211279\": rpc error: code = NotFound desc = could not find container \"f0ba85d260ae029a79e4a49563de14c31b434cec5f895de617ac35cbe1211279\": container with ID starting with f0ba85d260ae029a79e4a49563de14c31b434cec5f895de617ac35cbe1211279 not found: ID does not exist"
Nov 02 12:59:17.967179 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:17.967087    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"]
Nov 02 12:59:18.017762 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:18.017731    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-85b7x8b74"]
Nov 02 12:59:19.581297 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:19.581263    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4934733e-ba9a-42a0-8a48-ee258fafea16" path="/var/lib/kubelet/pods/4934733e-ba9a-42a0-8a48-ee258fafea16/volumes"
Nov 02 12:59:20.627277 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:20.627242    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e32b65_70ea_42cd_aa32_dbd4bd704fb9.slice/crio-conmon-f5a033c18ae532cf891832d54711982c49a61bef8967abfdf1e580677a238d49.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:59:22.774337 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:22.774299    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc" podUID="79bad023-80b3-4fad-9359-44639b61a21a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:59:22.969949 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:22.969915    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e32b65_70ea_42cd_aa32_dbd4bd704fb9.slice/crio-conmon-f5a033c18ae532cf891832d54711982c49a61bef8967abfdf1e580677a238d49.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:59:25.879881 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:25.879844    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:59:25.926774 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:25.926731    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e32b65_70ea_42cd_aa32_dbd4bd704fb9.slice/crio-conmon-f5a033c18ae532cf891832d54711982c49a61bef8967abfdf1e580677a238d49.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:59:25.926997 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:25.926741    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e32b65_70ea_42cd_aa32_dbd4bd704fb9.slice/crio-conmon-f5a033c18ae532cf891832d54711982c49a61bef8967abfdf1e580677a238d49.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:59:25.950037 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:25.949998    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"]
Nov 02 12:59:25.951421 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:25.951387    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc" podUID="79bad023-80b3-4fad-9359-44639b61a21a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 12:59:26.000016 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:25.999954    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"]
Nov 02 12:59:28.291166 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:28.291124    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:59:30.643160 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:30.643122    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e32b65_70ea_42cd_aa32_dbd4bd704fb9.slice/crio-conmon-f5a033c18ae532cf891832d54711982c49a61bef8967abfdf1e580677a238d49.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:59:30.950622 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:30.950533    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc" podUID="79bad023-80b3-4fad-9359-44639b61a21a" containerName="payload-processing" containerID="cri-o://2b6ca6e8c7c4442ab81c0462cc3ec254351d96f8943b63f1512ca4d2bbc099a1" gracePeriod=25
Nov 02 12:59:31.000362 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.000331    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv" podUID="49081fd3-bad5-43ac-b28d-1b9604a60bc1" containerName="payload-processing" containerID="cri-o://179203cebe26d69237a8393aec8f020f006fa16401523c049920a1f3ef9b1771" gracePeriod=25
Nov 02 12:59:31.255696 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.255672    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"
Nov 02 12:59:31.278321 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.278295    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:59:31.341520 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.341483    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49081fd3-bad5-43ac-b28d-1b9604a60bc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49081fd3-bad5-43ac-b28d-1b9604a60bc1-plugins-config-volume\") pod \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\" (UID: \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\") "
Nov 02 12:59:31.341710 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.341531    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79bad023-80b3-4fad-9359-44639b61a21a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79bad023-80b3-4fad-9359-44639b61a21a-routing-overlay-volume\") pod \"79bad023-80b3-4fad-9359-44639b61a21a\" (UID: \"79bad023-80b3-4fad-9359-44639b61a21a\") "
Nov 02 12:59:31.341710 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.341597    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49081fd3-bad5-43ac-b28d-1b9604a60bc1-kube-api-access-ft9zd\" (UniqueName: \"kubernetes.io/projected/49081fd3-bad5-43ac-b28d-1b9604a60bc1-kube-api-access-ft9zd\") pod \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\" (UID: \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\") "
Nov 02 12:59:31.341710 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.341659    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49081fd3-bad5-43ac-b28d-1b9604a60bc1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/49081fd3-bad5-43ac-b28d-1b9604a60bc1-provider-credentials\") pod \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\" (UID: \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\") "
Nov 02 12:59:31.341710 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.341685    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79bad023-80b3-4fad-9359-44639b61a21a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/79bad023-80b3-4fad-9359-44639b61a21a-provider-credentials\") pod \"79bad023-80b3-4fad-9359-44639b61a21a\" (UID: \"79bad023-80b3-4fad-9359-44639b61a21a\") "
Nov 02 12:59:31.341943 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.341734    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49081fd3-bad5-43ac-b28d-1b9604a60bc1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49081fd3-bad5-43ac-b28d-1b9604a60bc1-routing-overlay-volume\") pod \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\" (UID: \"49081fd3-bad5-43ac-b28d-1b9604a60bc1\") "
Nov 02 12:59:31.341943 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.341766    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79bad023-80b3-4fad-9359-44639b61a21a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bad023-80b3-4fad-9359-44639b61a21a-plugins-config-volume\") pod \"79bad023-80b3-4fad-9359-44639b61a21a\" (UID: \"79bad023-80b3-4fad-9359-44639b61a21a\") "
Nov 02 12:59:31.341943 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.341794    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79bad023-80b3-4fad-9359-44639b61a21a-kube-api-access-b8xk8\" (UniqueName: \"kubernetes.io/projected/79bad023-80b3-4fad-9359-44639b61a21a-kube-api-access-b8xk8\") pod \"79bad023-80b3-4fad-9359-44639b61a21a\" (UID: \"79bad023-80b3-4fad-9359-44639b61a21a\") "
Nov 02 12:59:31.341943 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.341927    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79bad023-80b3-4fad-9359-44639b61a21a-routing-overlay-volume" pod "79bad023-80b3-4fad-9359-44639b61a21a" (UID: "79bad023-80b3-4fad-9359-44639b61a21a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:31.342220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.341937    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49081fd3-bad5-43ac-b28d-1b9604a60bc1-plugins-config-volume" pod "49081fd3-bad5-43ac-b28d-1b9604a60bc1" (UID: "49081fd3-bad5-43ac-b28d-1b9604a60bc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:31.342220 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.342181    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49081fd3-bad5-43ac-b28d-1b9604a60bc1-routing-overlay-volume" pod "49081fd3-bad5-43ac-b28d-1b9604a60bc1" (UID: "49081fd3-bad5-43ac-b28d-1b9604a60bc1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:31.342352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.342327    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49081fd3-bad5-43ac-b28d-1b9604a60bc1-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:31.342413 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.342350    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79bad023-80b3-4fad-9359-44639b61a21a-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:31.342413 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.342366    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49081fd3-bad5-43ac-b28d-1b9604a60bc1-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:31.342502 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.342445    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79bad023-80b3-4fad-9359-44639b61a21a-plugins-config-volume" pod "79bad023-80b3-4fad-9359-44639b61a21a" (UID: "79bad023-80b3-4fad-9359-44639b61a21a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:31.343955 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.343936    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bad023-80b3-4fad-9359-44639b61a21a-kube-api-access-b8xk8" pod "79bad023-80b3-4fad-9359-44639b61a21a" (UID: "79bad023-80b3-4fad-9359-44639b61a21a"). InnerVolumeSpecName "kube-api-access-b8xk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:31.344047 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.343965    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49081fd3-bad5-43ac-b28d-1b9604a60bc1-provider-credentials" pod "49081fd3-bad5-43ac-b28d-1b9604a60bc1" (UID: "49081fd3-bad5-43ac-b28d-1b9604a60bc1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:31.344243 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.344224    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49081fd3-bad5-43ac-b28d-1b9604a60bc1-kube-api-access-ft9zd" pod "49081fd3-bad5-43ac-b28d-1b9604a60bc1" (UID: "49081fd3-bad5-43ac-b28d-1b9604a60bc1"). InnerVolumeSpecName "kube-api-access-ft9zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:31.344285 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.344235    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bad023-80b3-4fad-9359-44639b61a21a-provider-credentials" pod "79bad023-80b3-4fad-9359-44639b61a21a" (UID: "79bad023-80b3-4fad-9359-44639b61a21a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:31.442952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.442922    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft9zd\" (UniqueName: \"kubernetes.io/projected/49081fd3-bad5-43ac-b28d-1b9604a60bc1-kube-api-access-ft9zd\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:31.442952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.442949    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49081fd3-bad5-43ac-b28d-1b9604a60bc1-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:31.442952 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.442959    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79bad023-80b3-4fad-9359-44639b61a21a-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:31.443208 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.442970    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79bad023-80b3-4fad-9359-44639b61a21a-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:31.443208 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.442998    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8xk8\" (UniqueName: \"kubernetes.io/projected/79bad023-80b3-4fad-9359-44639b61a21a-kube-api-access-b8xk8\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 12:59:31.873524 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.873439    4464 generic.go:356] "Generic (PLEG): container finished" podID="49081fd3-bad5-43ac-b28d-1b9604a60bc1" containerID="179203cebe26d69237a8393aec8f020f006fa16401523c049920a1f3ef9b1771" exitCode=0
Nov 02 12:59:31.873524 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.873487    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"
Nov 02 12:59:31.874093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.873528    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv" event={"ID":"49081fd3-bad5-43ac-b28d-1b9604a60bc1","Type":"ContainerDied","Data":"179203cebe26d69237a8393aec8f020f006fa16401523c049920a1f3ef9b1771"}
Nov 02 12:59:31.874093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.873584    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv" event={"ID":"49081fd3-bad5-43ac-b28d-1b9604a60bc1","Type":"ContainerDied","Data":"b0b186d13b681b619e71886d44e29803a668c8ff7f66ec8dd69c33da63b844d0"}
Nov 02 12:59:31.874093 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.873625    4464 scope.go:122] "RemoveContainer" containerID="179203cebe26d69237a8393aec8f020f006fa16401523c049920a1f3ef9b1771"
Nov 02 12:59:31.875057 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.875030    4464 generic.go:356] "Generic (PLEG): container finished" podID="79bad023-80b3-4fad-9359-44639b61a21a" containerID="2b6ca6e8c7c4442ab81c0462cc3ec254351d96f8943b63f1512ca4d2bbc099a1" exitCode=0
Nov 02 12:59:31.875182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.875073    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"
Nov 02 12:59:31.875182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.875093    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc" event={"ID":"79bad023-80b3-4fad-9359-44639b61a21a","Type":"ContainerDied","Data":"2b6ca6e8c7c4442ab81c0462cc3ec254351d96f8943b63f1512ca4d2bbc099a1"}
Nov 02 12:59:31.875182 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.875135    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc" event={"ID":"79bad023-80b3-4fad-9359-44639b61a21a","Type":"ContainerDied","Data":"0bf9d40236657611e37d469e77944cf7bac55ee835d247b108c6bc352a967b94"}
Nov 02 12:59:31.882733 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.882710    4464 scope.go:122] "RemoveContainer" containerID="179203cebe26d69237a8393aec8f020f006fa16401523c049920a1f3ef9b1771"
Nov 02 12:59:31.883006 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:31.882962    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"179203cebe26d69237a8393aec8f020f006fa16401523c049920a1f3ef9b1771\": container with ID starting with 179203cebe26d69237a8393aec8f020f006fa16401523c049920a1f3ef9b1771 not found: ID does not exist" containerID="179203cebe26d69237a8393aec8f020f006fa16401523c049920a1f3ef9b1771"
Nov 02 12:59:31.883160 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.883015    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179203cebe26d69237a8393aec8f020f006fa16401523c049920a1f3ef9b1771"} err="failed to get container status \"179203cebe26d69237a8393aec8f020f006fa16401523c049920a1f3ef9b1771\": rpc error: code = NotFound desc = could not find container \"179203cebe26d69237a8393aec8f020f006fa16401523c049920a1f3ef9b1771\": container with ID starting with 179203cebe26d69237a8393aec8f020f006fa16401523c049920a1f3ef9b1771 not found: ID does not exist"
Nov 02 12:59:31.883160 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.883041    4464 scope.go:122] "RemoveContainer" containerID="2b6ca6e8c7c4442ab81c0462cc3ec254351d96f8943b63f1512ca4d2bbc099a1"
Nov 02 12:59:31.890963 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.890936    4464 scope.go:122] "RemoveContainer" containerID="2b6ca6e8c7c4442ab81c0462cc3ec254351d96f8943b63f1512ca4d2bbc099a1"
Nov 02 12:59:31.891255 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:31.891235    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6ca6e8c7c4442ab81c0462cc3ec254351d96f8943b63f1512ca4d2bbc099a1\": container with ID starting with 2b6ca6e8c7c4442ab81c0462cc3ec254351d96f8943b63f1512ca4d2bbc099a1 not found: ID does not exist" containerID="2b6ca6e8c7c4442ab81c0462cc3ec254351d96f8943b63f1512ca4d2bbc099a1"
Nov 02 12:59:31.891307 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:31.891262    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6ca6e8c7c4442ab81c0462cc3ec254351d96f8943b63f1512ca4d2bbc099a1"} err="failed to get container status \"2b6ca6e8c7c4442ab81c0462cc3ec254351d96f8943b63f1512ca4d2bbc099a1\": rpc error: code = NotFound desc = could not find container \"2b6ca6e8c7c4442ab81c0462cc3ec254351d96f8943b63f1512ca4d2bbc099a1\": container with ID starting with 2b6ca6e8c7c4442ab81c0462cc3ec254351d96f8943b63f1512ca4d2bbc099a1 not found: ID does not exist"
Nov 02 12:59:32.044263 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:32.044225    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"]
Nov 02 12:59:32.050253 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:32.050222    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5846gtdtv"]
Nov 02 12:59:32.162572 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:32.162481    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"]
Nov 02 12:59:32.167875 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:32.167849    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-5987tfzzc"]
Nov 02 12:59:33.581216 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:33.581185    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49081fd3-bad5-43ac-b28d-1b9604a60bc1" path="/var/lib/kubelet/pods/49081fd3-bad5-43ac-b28d-1b9604a60bc1/volumes"
Nov 02 12:59:33.581606 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:33.581551    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79bad023-80b3-4fad-9359-44639b61a21a" path="/var/lib/kubelet/pods/79bad023-80b3-4fad-9359-44639b61a21a/volumes"
Nov 02 12:59:38.004602 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:38.004563    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e32b65_70ea_42cd_aa32_dbd4bd704fb9.slice/crio-conmon-f5a033c18ae532cf891832d54711982c49a61bef8967abfdf1e580677a238d49.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:59:38.192078 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:38.192044    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:59:40.654158 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:40.654121    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e32b65_70ea_42cd_aa32_dbd4bd704fb9.slice/crio-conmon-f5a033c18ae532cf891832d54711982c49a61bef8967abfdf1e580677a238d49.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:59:47.269296 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.269259    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-978854467-vq6wf"]
Nov 02 12:59:47.269766 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.269749    4464 cpu_manager.go:414] "containerMap: removing container" podUID="49081fd3-bad5-43ac-b28d-1b9604a60bc1" containerName="payload-processing"
Nov 02 12:59:47.269766 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.269761    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49081fd3-bad5-43ac-b28d-1b9604a60bc1" containerName="payload-processing"
Nov 02 12:59:47.269842 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.269780    4464 cpu_manager.go:414] "containerMap: removing container" podUID="df38a272-fe8a-4cb1-a31b-2a37ca1f906d" containerName="payload-processing"
Nov 02 12:59:47.269842 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.269786    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df38a272-fe8a-4cb1-a31b-2a37ca1f906d" containerName="payload-processing"
Nov 02 12:59:47.269842 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.269800    4464 cpu_manager.go:414] "containerMap: removing container" podUID="40e32b65-70ea-42cd-aa32-dbd4bd704fb9" containerName="revoke-keys"
Nov 02 12:59:47.269842 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.269806    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e32b65-70ea-42cd-aa32-dbd4bd704fb9" containerName="revoke-keys"
Nov 02 12:59:47.269842 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.269813    4464 cpu_manager.go:414] "containerMap: removing container" podUID="4934733e-ba9a-42a0-8a48-ee258fafea16" containerName="payload-processing"
Nov 02 12:59:47.269842 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.269818    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4934733e-ba9a-42a0-8a48-ee258fafea16" containerName="payload-processing"
Nov 02 12:59:47.269842 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.269837    4464 cpu_manager.go:414] "containerMap: removing container" podUID="79bad023-80b3-4fad-9359-44639b61a21a" containerName="payload-processing"
Nov 02 12:59:47.269842 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.269842    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79bad023-80b3-4fad-9359-44639b61a21a" containerName="payload-processing"
Nov 02 12:59:47.270115 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.269916    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e32b65-70ea-42cd-aa32-dbd4bd704fb9" containerName="revoke-keys"
Nov 02 12:59:47.270115 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.269926    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="4934733e-ba9a-42a0-8a48-ee258fafea16" containerName="payload-processing"
Nov 02 12:59:47.270115 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.269941    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="79bad023-80b3-4fad-9359-44639b61a21a" containerName="payload-processing"
Nov 02 12:59:47.270115 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.269951    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="49081fd3-bad5-43ac-b28d-1b9604a60bc1" containerName="payload-processing"
Nov 02 12:59:47.270115 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.269959    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="df38a272-fe8a-4cb1-a31b-2a37ca1f906d" containerName="payload-processing"
Nov 02 12:59:47.274517 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.274501    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 12:59:47.329545 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.329517    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-018f0c4ba963\""
Nov 02 12:59:47.379300 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.379257    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 12:59:47.379300 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.379256    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 12:59:47.379300 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.379268    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 12:59:47.379546 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.379330    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-kr7r4\""
Nov 02 12:59:47.379546 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.379382    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 12:59:47.382124 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.382101    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-978854467-vq6wf"]
Nov 02 12:59:47.392212 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.392188    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53a6e480-9610-45ac-9e6a-3b5658f56f6c-routing-overlay-volume\") pod \"payload-processing-external-model-978854467-vq6wf\" (UID: \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\") " pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 12:59:47.392352 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.392259    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a6e480-9610-45ac-9e6a-3b5658f56f6c-plugins-config-volume\") pod \"payload-processing-external-model-978854467-vq6wf\" (UID: \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\") " pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 12:59:47.392423 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.392346    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69z6p\" (UniqueName: \"kubernetes.io/projected/53a6e480-9610-45ac-9e6a-3b5658f56f6c-kube-api-access-69z6p\") pod \"payload-processing-external-model-978854467-vq6wf\" (UID: \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\") " pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 12:59:47.392423 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.392415    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53a6e480-9610-45ac-9e6a-3b5658f56f6c-provider-credentials\") pod \"payload-processing-external-model-978854467-vq6wf\" (UID: \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\") " pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 12:59:47.493031 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.492994    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53a6e480-9610-45ac-9e6a-3b5658f56f6c-routing-overlay-volume\") pod \"payload-processing-external-model-978854467-vq6wf\" (UID: \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\") " pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 12:59:47.493216 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.493130    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a6e480-9610-45ac-9e6a-3b5658f56f6c-plugins-config-volume\") pod \"payload-processing-external-model-978854467-vq6wf\" (UID: \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\") " pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 12:59:47.493216 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.493183    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69z6p\" (UniqueName: \"kubernetes.io/projected/53a6e480-9610-45ac-9e6a-3b5658f56f6c-kube-api-access-69z6p\") pod \"payload-processing-external-model-978854467-vq6wf\" (UID: \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\") " pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 12:59:47.493355 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.493239    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53a6e480-9610-45ac-9e6a-3b5658f56f6c-provider-credentials\") pod \"payload-processing-external-model-978854467-vq6wf\" (UID: \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\") " pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 12:59:47.493709 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.493685    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53a6e480-9610-45ac-9e6a-3b5658f56f6c-routing-overlay-volume\") pod \"payload-processing-external-model-978854467-vq6wf\" (UID: \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\") " pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 12:59:47.493811 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.493799    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a6e480-9610-45ac-9e6a-3b5658f56f6c-plugins-config-volume\") pod \"payload-processing-external-model-978854467-vq6wf\" (UID: \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\") " pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 12:59:47.495512 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.495496    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53a6e480-9610-45ac-9e6a-3b5658f56f6c-provider-credentials\") pod \"payload-processing-external-model-978854467-vq6wf\" (UID: \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\") " pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 12:59:47.554817 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.554752    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69z6p\" (UniqueName: \"kubernetes.io/projected/53a6e480-9610-45ac-9e6a-3b5658f56f6c-kube-api-access-69z6p\") pod \"payload-processing-external-model-978854467-vq6wf\" (UID: \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\") " pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 12:59:47.587267 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.587244    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 12:59:47.958291 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:47.958264    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-978854467-vq6wf"]
Nov 02 12:59:47.959865 ip-10-0-133-239 kubenswrapper[4464]: W1102 12:59:47.959840    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a6e480_9610_45ac_9e6a_3b5658f56f6c.slice/crio-3cb25768d20af77a1667481e9998dcbd95b0f8ca4f3389f36e54ad10af1efac7 WatchSource:0}: Error finding container 3cb25768d20af77a1667481e9998dcbd95b0f8ca4f3389f36e54ad10af1efac7: Status 404 returned error can't find the container with id 3cb25768d20af77a1667481e9998dcbd95b0f8ca4f3389f36e54ad10af1efac7
Nov 02 12:59:48.965241 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:48.965201    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf" event={"ID":"53a6e480-9610-45ac-9e6a-3b5658f56f6c","Type":"ContainerStarted","Data":"de8f95e850215850da909a029a04696107e6eb2786e7f89bfc6fd2d7d51f21fe"}
Nov 02 12:59:48.965637 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:48.965292    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 12:59:48.965637 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:48.965311    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf" event={"ID":"53a6e480-9610-45ac-9e6a-3b5658f56f6c","Type":"ContainerStarted","Data":"3cb25768d20af77a1667481e9998dcbd95b0f8ca4f3389f36e54ad10af1efac7"}
Nov 02 12:59:49.084929 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:49.084876    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf" podStartSLOduration=2.084859317 podStartE2EDuration="2.084859317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:59:49.083706906 +0000 UTC m=+3817.647415317" watchObservedRunningTime="2026-11-02 12:59:49.084859317 +0000 UTC m=+3817.648567723"
Nov 02 12:59:50.665133 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:50.665096    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e32b65_70ea_42cd_aa32_dbd4bd704fb9.slice/crio-conmon-f5a033c18ae532cf891832d54711982c49a61bef8967abfdf1e580677a238d49.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:59:51.109113 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:51.109073    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 12:59:53.011081 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:53.011044    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e32b65_70ea_42cd_aa32_dbd4bd704fb9.slice/crio-conmon-f5a033c18ae532cf891832d54711982c49a61bef8967abfdf1e580677a238d49.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:59:58.292194 ip-10-0-133-239 kubenswrapper[4464]: E1102 12:59:58.292117    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 12:59:59.974161 ip-10-0-133-239 kubenswrapper[4464]: I1102 12:59:59.974127    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 13:00:00.172329 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.171256    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2"]
Nov 02 13:00:00.177118 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.177084    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2"
Nov 02 13:00:00.178525 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.178495    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7"]
Nov 02 13:00:00.182012 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.181991    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7"
Nov 02 13:00:00.230522 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.230444    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2"]
Nov 02 13:00:00.230781 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.230553    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:00:00.230781 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.230557    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 13:00:00.230781 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.230666    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:00:00.233809 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.233784    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7"]
Nov 02 13:00:00.317232 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.317195    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttzb9\" (UniqueName: \"kubernetes.io/projected/11f6e9ac-1ad6-4ea2-a1a5-a251d6136191-kube-api-access-ttzb9\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2\" (UID: \"11f6e9ac-1ad6-4ea2-a1a5-a251d6136191\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2"
Nov 02 13:00:00.317430 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.317315    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4vdw\" (UniqueName: \"kubernetes.io/projected/8ff9fea6-3a26-4908-873a-22c20d43673d-kube-api-access-k4vdw\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7\" (UID: \"8ff9fea6-3a26-4908-873a-22c20d43673d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7"
Nov 02 13:00:00.317430 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.317347    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff9fea6-3a26-4908-873a-22c20d43673d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7\" (UID: \"8ff9fea6-3a26-4908-873a-22c20d43673d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7"
Nov 02 13:00:00.317562 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.317510    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f6e9ac-1ad6-4ea2-a1a5-a251d6136191-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2\" (UID: \"11f6e9ac-1ad6-4ea2-a1a5-a251d6136191\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2"
Nov 02 13:00:00.419020 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.418956    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4vdw\" (UniqueName: \"kubernetes.io/projected/8ff9fea6-3a26-4908-873a-22c20d43673d-kube-api-access-k4vdw\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7\" (UID: \"8ff9fea6-3a26-4908-873a-22c20d43673d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7"
Nov 02 13:00:00.419238 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.419055    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff9fea6-3a26-4908-873a-22c20d43673d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7\" (UID: \"8ff9fea6-3a26-4908-873a-22c20d43673d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7"
Nov 02 13:00:00.419238 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.419156    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f6e9ac-1ad6-4ea2-a1a5-a251d6136191-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2\" (UID: \"11f6e9ac-1ad6-4ea2-a1a5-a251d6136191\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2"
Nov 02 13:00:00.419238 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.419183    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttzb9\" (UniqueName: \"kubernetes.io/projected/11f6e9ac-1ad6-4ea2-a1a5-a251d6136191-kube-api-access-ttzb9\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2\" (UID: \"11f6e9ac-1ad6-4ea2-a1a5-a251d6136191\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2"
Nov 02 13:00:00.419907 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.419883    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff9fea6-3a26-4908-873a-22c20d43673d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7\" (UID: \"8ff9fea6-3a26-4908-873a-22c20d43673d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7"
Nov 02 13:00:00.419962 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.419883    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f6e9ac-1ad6-4ea2-a1a5-a251d6136191-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2\" (UID: \"11f6e9ac-1ad6-4ea2-a1a5-a251d6136191\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2"
Nov 02 13:00:00.481586 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.481508    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4vdw\" (UniqueName: \"kubernetes.io/projected/8ff9fea6-3a26-4908-873a-22c20d43673d-kube-api-access-k4vdw\") pod \"maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7\" (UID: \"8ff9fea6-3a26-4908-873a-22c20d43673d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7"
Nov 02 13:00:00.481586 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.481507    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttzb9\" (UniqueName: \"kubernetes.io/projected/11f6e9ac-1ad6-4ea2-a1a5-a251d6136191-kube-api-access-ttzb9\") pod \"maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2\" (UID: \"11f6e9ac-1ad6-4ea2-a1a5-a251d6136191\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2"
Nov 02 13:00:00.492141 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.492117    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2"
Nov 02 13:00:00.497405 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.497381    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7"
Nov 02 13:00:00.717597 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:00.717566    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e32b65_70ea_42cd_aa32_dbd4bd704fb9.slice/crio-conmon-f5a033c18ae532cf891832d54711982c49a61bef8967abfdf1e580677a238d49.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 13:00:00.869867 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.869842    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2"]
Nov 02 13:00:00.871896 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:00:00.871862    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f6e9ac_1ad6_4ea2_a1a5_a251d6136191.slice/crio-707991703c8017d8fe4ea7b30a8f03bc48fcfa56da640c20dd54cf40f1b21ef9 WatchSource:0}: Error finding container 707991703c8017d8fe4ea7b30a8f03bc48fcfa56da640c20dd54cf40f1b21ef9: Status 404 returned error can't find the container with id 707991703c8017d8fe4ea7b30a8f03bc48fcfa56da640c20dd54cf40f1b21ef9
Nov 02 13:00:00.918697 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:00.918675    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7"]
Nov 02 13:00:00.920036 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:00:00.920013    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff9fea6_3a26_4908_873a_22c20d43673d.slice/crio-702b3c2cc4456b19faa980b6d3c0dd3cce1127237d996175ba1d5171fb284f79 WatchSource:0}: Error finding container 702b3c2cc4456b19faa980b6d3c0dd3cce1127237d996175ba1d5171fb284f79: Status 404 returned error can't find the container with id 702b3c2cc4456b19faa980b6d3c0dd3cce1127237d996175ba1d5171fb284f79
Nov 02 13:00:01.026228 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:01.026195    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2" event={"ID":"11f6e9ac-1ad6-4ea2-a1a5-a251d6136191","Type":"ContainerStarted","Data":"707991703c8017d8fe4ea7b30a8f03bc48fcfa56da640c20dd54cf40f1b21ef9"}
Nov 02 13:00:01.027354 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:01.027331    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7" event={"ID":"8ff9fea6-3a26-4908-873a-22c20d43673d","Type":"ContainerStarted","Data":"702b3c2cc4456b19faa980b6d3c0dd3cce1127237d996175ba1d5171fb284f79"}
Nov 02 13:00:02.034492 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:02.034458    4464 generic.go:356] "Generic (PLEG): container finished" podID="11f6e9ac-1ad6-4ea2-a1a5-a251d6136191" containerID="53a1140ecf06e880138e9665238218ac239d744bbecb670a42ab1bc2492b894b" exitCode=60
Nov 02 13:00:02.035013 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:02.034544    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2" event={"ID":"11f6e9ac-1ad6-4ea2-a1a5-a251d6136191","Type":"ContainerDied","Data":"53a1140ecf06e880138e9665238218ac239d744bbecb670a42ab1bc2492b894b"}
Nov 02 13:00:02.035013 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:02.034774    4464 scope.go:122] "RemoveContainer" containerID="53a1140ecf06e880138e9665238218ac239d744bbecb670a42ab1bc2492b894b"
Nov 02 13:00:02.036294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:02.036269    4464 generic.go:356] "Generic (PLEG): container finished" podID="8ff9fea6-3a26-4908-873a-22c20d43673d" containerID="66feda820d3f4a9b51fed2a40be7ac1db38f25eb3a3ea473211c773c93d859e7" exitCode=60
Nov 02 13:00:02.036389 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:02.036300    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7" event={"ID":"8ff9fea6-3a26-4908-873a-22c20d43673d","Type":"ContainerDied","Data":"66feda820d3f4a9b51fed2a40be7ac1db38f25eb3a3ea473211c773c93d859e7"}
Nov 02 13:00:02.036559 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:02.036543    4464 scope.go:122] "RemoveContainer" containerID="66feda820d3f4a9b51fed2a40be7ac1db38f25eb3a3ea473211c773c93d859e7"
Nov 02 13:00:03.043781 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:03.043704    4464 generic.go:356] "Generic (PLEG): container finished" podID="11f6e9ac-1ad6-4ea2-a1a5-a251d6136191" containerID="d64864c0da5db2bf7ea42e05d8c80df892142c38427fa5d4d668876399f662a1" exitCode=60
Nov 02 13:00:03.044199 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:03.043793    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2" event={"ID":"11f6e9ac-1ad6-4ea2-a1a5-a251d6136191","Type":"ContainerDied","Data":"d64864c0da5db2bf7ea42e05d8c80df892142c38427fa5d4d668876399f662a1"}
Nov 02 13:00:03.044199 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:03.043850    4464 scope.go:122] "RemoveContainer" containerID="53a1140ecf06e880138e9665238218ac239d744bbecb670a42ab1bc2492b894b"
Nov 02 13:00:03.044199 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:03.044033    4464 scope.go:122] "RemoveContainer" containerID="d64864c0da5db2bf7ea42e05d8c80df892142c38427fa5d4d668876399f662a1"
Nov 02 13:00:03.044354 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:03.044295    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2_odh-ai-gateway-infra(11f6e9ac-1ad6-4ea2-a1a5-a251d6136191)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2" podUID="11f6e9ac-1ad6-4ea2-a1a5-a251d6136191"
Nov 02 13:00:03.045811 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:03.045791    4464 generic.go:356] "Generic (PLEG): container finished" podID="8ff9fea6-3a26-4908-873a-22c20d43673d" containerID="25657ba2971aef2727268150c903c66f0314a9ff2ac77c4df5b968b1374c857a" exitCode=60
Nov 02 13:00:03.045906 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:03.045856    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7" event={"ID":"8ff9fea6-3a26-4908-873a-22c20d43673d","Type":"ContainerDied","Data":"25657ba2971aef2727268150c903c66f0314a9ff2ac77c4df5b968b1374c857a"}
Nov 02 13:00:03.046059 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:03.046046    4464 scope.go:122] "RemoveContainer" containerID="25657ba2971aef2727268150c903c66f0314a9ff2ac77c4df5b968b1374c857a"
Nov 02 13:00:03.046230 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:03.046217    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7_odh-ai-gateway-infra(8ff9fea6-3a26-4908-873a-22c20d43673d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7" podUID="8ff9fea6-3a26-4908-873a-22c20d43673d"
Nov 02 13:00:03.054940 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:03.054926    4464 scope.go:122] "RemoveContainer" containerID="66feda820d3f4a9b51fed2a40be7ac1db38f25eb3a3ea473211c773c93d859e7"
Nov 02 13:00:04.052390 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:04.052363    4464 scope.go:122] "RemoveContainer" containerID="d64864c0da5db2bf7ea42e05d8c80df892142c38427fa5d4d668876399f662a1"
Nov 02 13:00:04.052825 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:04.052593    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2_odh-ai-gateway-infra(11f6e9ac-1ad6-4ea2-a1a5-a251d6136191)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2" podUID="11f6e9ac-1ad6-4ea2-a1a5-a251d6136191"
Nov 02 13:00:04.053921 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:04.053901    4464 scope.go:122] "RemoveContainer" containerID="25657ba2971aef2727268150c903c66f0314a9ff2ac77c4df5b968b1374c857a"
Nov 02 13:00:04.054103 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:04.054085    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7_odh-ai-gateway-infra(8ff9fea6-3a26-4908-873a-22c20d43673d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7" podUID="8ff9fea6-3a26-4908-873a-22c20d43673d"
Nov 02 13:00:06.498753 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:06.498721    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:00:08.010858 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:08.010827    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e32b65_70ea_42cd_aa32_dbd4bd704fb9.slice/crio-conmon-f5a033c18ae532cf891832d54711982c49a61bef8967abfdf1e580677a238d49.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 13:00:10.728829 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:10.728790    4464 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e32b65_70ea_42cd_aa32_dbd4bd704fb9.slice/crio-conmon-f5a033c18ae532cf891832d54711982c49a61bef8967abfdf1e580677a238d49.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 13:00:13.575609 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:13.575578    4464 scope.go:122] "RemoveContainer" containerID="d64864c0da5db2bf7ea42e05d8c80df892142c38427fa5d4d668876399f662a1"
Nov 02 13:00:14.107430 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:14.107395    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2" event={"ID":"11f6e9ac-1ad6-4ea2-a1a5-a251d6136191","Type":"ContainerStarted","Data":"d354efef6735e397f307d1cdf6d21d20032218d1ed52435488fec6e628c9bdaf"}
Nov 02 13:00:14.225634 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:14.225584    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2" podStartSLOduration=14.225570605 podStartE2EDuration="14.225570605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:14.225058957 +0000 UTC m=+3842.788767367" watchObservedRunningTime="2026-11-02 13:00:14.225570605 +0000 UTC m=+3842.789279032"
Nov 02 13:00:15.114804 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:15.114768    4464 generic.go:356] "Generic (PLEG): container finished" podID="11f6e9ac-1ad6-4ea2-a1a5-a251d6136191" containerID="d354efef6735e397f307d1cdf6d21d20032218d1ed52435488fec6e628c9bdaf" exitCode=60
Nov 02 13:00:15.115323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:15.114840    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2" event={"ID":"11f6e9ac-1ad6-4ea2-a1a5-a251d6136191","Type":"ContainerDied","Data":"d354efef6735e397f307d1cdf6d21d20032218d1ed52435488fec6e628c9bdaf"}
Nov 02 13:00:15.115323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:15.114955    4464 scope.go:122] "RemoveContainer" containerID="d64864c0da5db2bf7ea42e05d8c80df892142c38427fa5d4d668876399f662a1"
Nov 02 13:00:15.115323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:15.115281    4464 scope.go:122] "RemoveContainer" containerID="d354efef6735e397f307d1cdf6d21d20032218d1ed52435488fec6e628c9bdaf"
Nov 02 13:00:15.115682 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:15.115525    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2_odh-ai-gateway-infra(11f6e9ac-1ad6-4ea2-a1a5-a251d6136191)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2" podUID="11f6e9ac-1ad6-4ea2-a1a5-a251d6136191"
Nov 02 13:00:15.241012 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:15.240962    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2"]
Nov 02 13:00:15.575119 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:15.575081    4464 scope.go:122] "RemoveContainer" containerID="25657ba2971aef2727268150c903c66f0314a9ff2ac77c4df5b968b1374c857a"
Nov 02 13:00:16.124776 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:16.124749    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7" event={"ID":"8ff9fea6-3a26-4908-873a-22c20d43673d","Type":"ContainerStarted","Data":"63332acff4b7ffca4489512f30dfbc08543410deadde4caee95c6f3b3ddacad0"}
Nov 02 13:00:16.267450 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:16.267424    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2"
Nov 02 13:00:16.360363 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:16.360310    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7" podStartSLOduration=16.36029445 podStartE2EDuration="16.36029445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:16.358183591 +0000 UTC m=+3844.921892005" watchObservedRunningTime="2026-11-02 13:00:16.36029445 +0000 UTC m=+3844.924002904"
Nov 02 13:00:16.360581 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:16.360545    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f6e9ac-1ad6-4ea2-a1a5-a251d6136191-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f6e9ac-1ad6-4ea2-a1a5-a251d6136191-maas-api-service-ca\") pod \"11f6e9ac-1ad6-4ea2-a1a5-a251d6136191\" (UID: \"11f6e9ac-1ad6-4ea2-a1a5-a251d6136191\") "
Nov 02 13:00:16.360691 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:16.360674    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f6e9ac-1ad6-4ea2-a1a5-a251d6136191-kube-api-access-ttzb9\" (UniqueName: \"kubernetes.io/projected/11f6e9ac-1ad6-4ea2-a1a5-a251d6136191-kube-api-access-ttzb9\") pod \"11f6e9ac-1ad6-4ea2-a1a5-a251d6136191\" (UID: \"11f6e9ac-1ad6-4ea2-a1a5-a251d6136191\") "
Nov 02 13:00:16.360896 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:16.360875    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f6e9ac-1ad6-4ea2-a1a5-a251d6136191-maas-api-service-ca" pod "11f6e9ac-1ad6-4ea2-a1a5-a251d6136191" (UID: "11f6e9ac-1ad6-4ea2-a1a5-a251d6136191"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:16.362764 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:16.362741    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f6e9ac-1ad6-4ea2-a1a5-a251d6136191-kube-api-access-ttzb9" pod "11f6e9ac-1ad6-4ea2-a1a5-a251d6136191" (UID: "11f6e9ac-1ad6-4ea2-a1a5-a251d6136191"). InnerVolumeSpecName "kube-api-access-ttzb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:16.461960 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:16.461887    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttzb9\" (UniqueName: \"kubernetes.io/projected/11f6e9ac-1ad6-4ea2-a1a5-a251d6136191-kube-api-access-ttzb9\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:16.461960 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:16.461910    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f6e9ac-1ad6-4ea2-a1a5-a251d6136191-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:17.131321 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:17.131283    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2"
Nov 02 13:00:17.131321 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:17.131300    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2" event={"ID":"11f6e9ac-1ad6-4ea2-a1a5-a251d6136191","Type":"ContainerDied","Data":"707991703c8017d8fe4ea7b30a8f03bc48fcfa56da640c20dd54cf40f1b21ef9"}
Nov 02 13:00:17.131901 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:17.131362    4464 scope.go:122] "RemoveContainer" containerID="d354efef6735e397f307d1cdf6d21d20032218d1ed52435488fec6e628c9bdaf"
Nov 02 13:00:17.133705 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:17.133679    4464 generic.go:356] "Generic (PLEG): container finished" podID="8ff9fea6-3a26-4908-873a-22c20d43673d" containerID="63332acff4b7ffca4489512f30dfbc08543410deadde4caee95c6f3b3ddacad0" exitCode=60
Nov 02 13:00:17.133804 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:17.133740    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7" event={"ID":"8ff9fea6-3a26-4908-873a-22c20d43673d","Type":"ContainerDied","Data":"63332acff4b7ffca4489512f30dfbc08543410deadde4caee95c6f3b3ddacad0"}
Nov 02 13:00:17.134110 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:17.134096    4464 scope.go:122] "RemoveContainer" containerID="63332acff4b7ffca4489512f30dfbc08543410deadde4caee95c6f3b3ddacad0"
Nov 02 13:00:17.134346 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:17.134328    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7_odh-ai-gateway-infra(8ff9fea6-3a26-4908-873a-22c20d43673d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7" podUID="8ff9fea6-3a26-4908-873a-22c20d43673d"
Nov 02 13:00:17.140777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:17.140762    4464 scope.go:122] "RemoveContainer" containerID="25657ba2971aef2727268150c903c66f0314a9ff2ac77c4df5b968b1374c857a"
Nov 02 13:00:17.298386 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:17.298350    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2"]
Nov 02 13:00:17.306585 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:17.306555    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-3a78b4-29893740-hlsq2"]
Nov 02 13:00:17.368592 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:17.368562    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7"]
Nov 02 13:00:17.581739 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:17.581706    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f6e9ac-1ad6-4ea2-a1a5-a251d6136191" path="/var/lib/kubelet/pods/11f6e9ac-1ad6-4ea2-a1a5-a251d6136191/volumes"
Nov 02 13:00:18.276719 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.276693    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7"
Nov 02 13:00:18.310602 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.310574    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"]
Nov 02 13:00:18.311052 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.311037    4464 cpu_manager.go:414] "containerMap: removing container" podUID="11f6e9ac-1ad6-4ea2-a1a5-a251d6136191" containerName="cleanup"
Nov 02 13:00:18.311125 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.311053    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f6e9ac-1ad6-4ea2-a1a5-a251d6136191" containerName="cleanup"
Nov 02 13:00:18.311125 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.311072    4464 cpu_manager.go:414] "containerMap: removing container" podUID="8ff9fea6-3a26-4908-873a-22c20d43673d" containerName="cleanup"
Nov 02 13:00:18.311125 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.311077    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff9fea6-3a26-4908-873a-22c20d43673d" containerName="cleanup"
Nov 02 13:00:18.311125 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.311086    4464 cpu_manager.go:414] "containerMap: removing container" podUID="8ff9fea6-3a26-4908-873a-22c20d43673d" containerName="cleanup"
Nov 02 13:00:18.311125 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.311090    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff9fea6-3a26-4908-873a-22c20d43673d" containerName="cleanup"
Nov 02 13:00:18.311125 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.311098    4464 cpu_manager.go:414] "containerMap: removing container" podUID="8ff9fea6-3a26-4908-873a-22c20d43673d" containerName="cleanup"
Nov 02 13:00:18.311125 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.311102    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff9fea6-3a26-4908-873a-22c20d43673d" containerName="cleanup"
Nov 02 13:00:18.311125 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.311109    4464 cpu_manager.go:414] "containerMap: removing container" podUID="11f6e9ac-1ad6-4ea2-a1a5-a251d6136191" containerName="cleanup"
Nov 02 13:00:18.311125 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.311113    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f6e9ac-1ad6-4ea2-a1a5-a251d6136191" containerName="cleanup"
Nov 02 13:00:18.311411 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.311192    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f6e9ac-1ad6-4ea2-a1a5-a251d6136191" containerName="cleanup"
Nov 02 13:00:18.311411 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.311204    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff9fea6-3a26-4908-873a-22c20d43673d" containerName="cleanup"
Nov 02 13:00:18.311411 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.311210    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff9fea6-3a26-4908-873a-22c20d43673d" containerName="cleanup"
Nov 02 13:00:18.311411 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.311215    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff9fea6-3a26-4908-873a-22c20d43673d" containerName="cleanup"
Nov 02 13:00:18.311411 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.311221    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f6e9ac-1ad6-4ea2-a1a5-a251d6136191" containerName="cleanup"
Nov 02 13:00:18.314349 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.314334    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"
Nov 02 13:00:18.368073 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.368041    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:00:18.371213 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.371187    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"]
Nov 02 13:00:18.380689 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.380663    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff9fea6-3a26-4908-873a-22c20d43673d-kube-api-access-k4vdw\" (UniqueName: \"kubernetes.io/projected/8ff9fea6-3a26-4908-873a-22c20d43673d-kube-api-access-k4vdw\") pod \"8ff9fea6-3a26-4908-873a-22c20d43673d\" (UID: \"8ff9fea6-3a26-4908-873a-22c20d43673d\") "
Nov 02 13:00:18.380849 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.380715    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ff9fea6-3a26-4908-873a-22c20d43673d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff9fea6-3a26-4908-873a-22c20d43673d-maas-api-service-ca\") pod \"8ff9fea6-3a26-4908-873a-22c20d43673d\" (UID: \"8ff9fea6-3a26-4908-873a-22c20d43673d\") "
Nov 02 13:00:18.380988 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.380957    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be60869f-1203-4f0f-aa25-1d6973eef5ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-564d99db-mk457\" (UID: \"be60869f-1203-4f0f-aa25-1d6973eef5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"
Nov 02 13:00:18.381100 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.381078    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff9fea6-3a26-4908-873a-22c20d43673d-maas-api-service-ca" pod "8ff9fea6-3a26-4908-873a-22c20d43673d" (UID: "8ff9fea6-3a26-4908-873a-22c20d43673d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:18.381160 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.381092    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be60869f-1203-4f0f-aa25-1d6973eef5ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-564d99db-mk457\" (UID: \"be60869f-1203-4f0f-aa25-1d6973eef5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"
Nov 02 13:00:18.381204 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.381175    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9547\" (UniqueName: \"kubernetes.io/projected/be60869f-1203-4f0f-aa25-1d6973eef5ef-kube-api-access-t9547\") pod \"maas-api-revoke-sub-564d99db-mk457\" (UID: \"be60869f-1203-4f0f-aa25-1d6973eef5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"
Nov 02 13:00:18.381340 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.381324    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff9fea6-3a26-4908-873a-22c20d43673d-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:18.382739 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.382720    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff9fea6-3a26-4908-873a-22c20d43673d-kube-api-access-k4vdw" pod "8ff9fea6-3a26-4908-873a-22c20d43673d" (UID: "8ff9fea6-3a26-4908-873a-22c20d43673d"). InnerVolumeSpecName "kube-api-access-k4vdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:18.482440 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.482404    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be60869f-1203-4f0f-aa25-1d6973eef5ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-564d99db-mk457\" (UID: \"be60869f-1203-4f0f-aa25-1d6973eef5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"
Nov 02 13:00:18.482440 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.482445    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9547\" (UniqueName: \"kubernetes.io/projected/be60869f-1203-4f0f-aa25-1d6973eef5ef-kube-api-access-t9547\") pod \"maas-api-revoke-sub-564d99db-mk457\" (UID: \"be60869f-1203-4f0f-aa25-1d6973eef5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"
Nov 02 13:00:18.482745 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.482506    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be60869f-1203-4f0f-aa25-1d6973eef5ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-564d99db-mk457\" (UID: \"be60869f-1203-4f0f-aa25-1d6973eef5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"
Nov 02 13:00:18.482745 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.482561    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4vdw\" (UniqueName: \"kubernetes.io/projected/8ff9fea6-3a26-4908-873a-22c20d43673d-kube-api-access-k4vdw\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:18.483215 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.483190    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be60869f-1203-4f0f-aa25-1d6973eef5ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-564d99db-mk457\" (UID: \"be60869f-1203-4f0f-aa25-1d6973eef5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"
Nov 02 13:00:18.484831 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.484811    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be60869f-1203-4f0f-aa25-1d6973eef5ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-564d99db-mk457\" (UID: \"be60869f-1203-4f0f-aa25-1d6973eef5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"
Nov 02 13:00:18.544599 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.544568    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9547\" (UniqueName: \"kubernetes.io/projected/be60869f-1203-4f0f-aa25-1d6973eef5ef-kube-api-access-t9547\") pod \"maas-api-revoke-sub-564d99db-mk457\" (UID: \"be60869f-1203-4f0f-aa25-1d6973eef5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"
Nov 02 13:00:18.626615 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.626578    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"
Nov 02 13:00:18.999695 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:18.999669    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"]
Nov 02 13:00:19.000429 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:00:19.000400    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe60869f_1203_4f0f_aa25_1d6973eef5ef.slice/crio-d7d726263b7269300fc20f04f9e501e1924b8de40996ec7fbd4a405af2c0660c WatchSource:0}: Error finding container d7d726263b7269300fc20f04f9e501e1924b8de40996ec7fbd4a405af2c0660c: Status 404 returned error can't find the container with id d7d726263b7269300fc20f04f9e501e1924b8de40996ec7fbd4a405af2c0660c
Nov 02 13:00:19.148442 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.148412    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7"
Nov 02 13:00:19.148442 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.148417    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7" event={"ID":"8ff9fea6-3a26-4908-873a-22c20d43673d","Type":"ContainerDied","Data":"702b3c2cc4456b19faa980b6d3c0dd3cce1127237d996175ba1d5171fb284f79"}
Nov 02 13:00:19.148687 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.148473    4464 scope.go:122] "RemoveContainer" containerID="63332acff4b7ffca4489512f30dfbc08543410deadde4caee95c6f3b3ddacad0"
Nov 02 13:00:19.150001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.149942    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457" event={"ID":"be60869f-1203-4f0f-aa25-1d6973eef5ef","Type":"ContainerStarted","Data":"d7d726263b7269300fc20f04f9e501e1924b8de40996ec7fbd4a405af2c0660c"}
Nov 02 13:00:19.157576 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.157552    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"]
Nov 02 13:00:19.158192 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.158170    4464 cpu_manager.go:414] "containerMap: removing container" podUID="11f6e9ac-1ad6-4ea2-a1a5-a251d6136191" containerName="cleanup"
Nov 02 13:00:19.158290 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.158194    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f6e9ac-1ad6-4ea2-a1a5-a251d6136191" containerName="cleanup"
Nov 02 13:00:19.158354 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.158347    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f6e9ac-1ad6-4ea2-a1a5-a251d6136191" containerName="cleanup"
Nov 02 13:00:19.163267 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.163247    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.189404 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.189382    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.189511 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.189421    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxzmm\" (UniqueName: \"kubernetes.io/projected/f6025f4f-a2b0-4537-8a34-439e39b04658-kube-api-access-kxzmm\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.189511 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.189445    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.189611 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.189595    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6025f4f-a2b0-4537-8a34-439e39b04658-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.189657 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.189644    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.189696 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.189677    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.189762 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.189738    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.189812 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.189773    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.189846 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.189812    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.219748 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.219380    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c5ldd\""
Nov 02 13:00:19.234250 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.233810    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"]
Nov 02 13:00:19.290523 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.290444    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6025f4f-a2b0-4537-8a34-439e39b04658-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.290523 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.290491    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.290523 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.290522    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.291193 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.290550    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.291193 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.290570    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.291193 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.290598    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.291193 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.290631    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.291193 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.290684    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxzmm\" (UniqueName: \"kubernetes.io/projected/f6025f4f-a2b0-4537-8a34-439e39b04658-kube-api-access-kxzmm\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.291193 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.290718    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.291193 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.290968    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.291193 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.291064    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.291193 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.291087    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.291704 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.291274    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6025f4f-a2b0-4537-8a34-439e39b04658-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.291704 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.291357    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.293137 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.293112    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.293255 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.293187    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.353117 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.353077    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.353257 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.353147    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxzmm\" (UniqueName: \"kubernetes.io/projected/f6025f4f-a2b0-4537-8a34-439e39b04658-kube-api-access-kxzmm\") pod \"maas-default-gateway-openshift-default-6dcf757899-r6j5r\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.438101 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.438068    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7"]
Nov 02 13:00:19.444652 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.444624    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6ac03f-29893740-6ppf7"]
Nov 02 13:00:19.482989 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.482947    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:19.582869 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.582799    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ff9fea6-3a26-4908-873a-22c20d43673d" path="/var/lib/kubelet/pods/8ff9fea6-3a26-4908-873a-22c20d43673d/volumes"
Nov 02 13:00:19.910812 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.910785    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"]
Nov 02 13:00:19.932861 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:00:19.932832    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6025f4f_a2b0_4537_8a34_439e39b04658.slice/crio-72ead62ac3e3e41dc6463d7b909dfe9facb2b40be5fd70630e4f83eb4975fd9b WatchSource:0}: Error finding container 72ead62ac3e3e41dc6463d7b909dfe9facb2b40be5fd70630e4f83eb4975fd9b: Status 404 returned error can't find the container with id 72ead62ac3e3e41dc6463d7b909dfe9facb2b40be5fd70630e4f83eb4975fd9b
Nov 02 13:00:19.935100 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.935069    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:00:19.935191 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.935140    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:00:19.935191 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:19.935166    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:00:20.157960 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:20.157870    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r" event={"ID":"f6025f4f-a2b0-4537-8a34-439e39b04658","Type":"ContainerStarted","Data":"a0a25fae4b32208e0c482fddced2b12d1ccd4ff505bc03b6556a0f10bee4c6e1"}
Nov 02 13:00:20.157960 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:20.157927    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r" event={"ID":"f6025f4f-a2b0-4537-8a34-439e39b04658","Type":"ContainerStarted","Data":"72ead62ac3e3e41dc6463d7b909dfe9facb2b40be5fd70630e4f83eb4975fd9b"}
Nov 02 13:00:20.159554 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:20.159527    4464 generic.go:356] "Generic (PLEG): container finished" podID="be60869f-1203-4f0f-aa25-1d6973eef5ef" containerID="110e78cc961fb23d6555c21871a1ba1bb31f5f944d8beb303e8bd9d4bbb1199b" exitCode=0
Nov 02 13:00:20.159708 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:20.159620    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457" event={"ID":"be60869f-1203-4f0f-aa25-1d6973eef5ef","Type":"ContainerDied","Data":"110e78cc961fb23d6555c21871a1ba1bb31f5f944d8beb303e8bd9d4bbb1199b"}
Nov 02 13:00:20.282676 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:20.282598    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r" podStartSLOduration=1.2825762680000001 podStartE2EDuration="1.282576268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:20.280348638 +0000 UTC m=+3848.844057057" watchObservedRunningTime="2026-11-02 13:00:20.282576268 +0000 UTC m=+3848.846284692"
Nov 02 13:00:20.484174 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:20.484127    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:20.486038 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:20.486006    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r6j5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:00:20.486165 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:20.486076    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r" podUID="f6025f4f-a2b0-4537-8a34-439e39b04658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 02 13:00:21.348736 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:21.348712    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"
Nov 02 13:00:21.413395 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:21.413366    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be60869f-1203-4f0f-aa25-1d6973eef5ef-kube-api-access-t9547\" (UniqueName: \"kubernetes.io/projected/be60869f-1203-4f0f-aa25-1d6973eef5ef-kube-api-access-t9547\") pod \"be60869f-1203-4f0f-aa25-1d6973eef5ef\" (UID: \"be60869f-1203-4f0f-aa25-1d6973eef5ef\") "
Nov 02 13:00:21.413606 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:21.413517    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be60869f-1203-4f0f-aa25-1d6973eef5ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be60869f-1203-4f0f-aa25-1d6973eef5ef-maas-api-openshift-ca\") pod \"be60869f-1203-4f0f-aa25-1d6973eef5ef\" (UID: \"be60869f-1203-4f0f-aa25-1d6973eef5ef\") "
Nov 02 13:00:21.413606 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:21.413554    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be60869f-1203-4f0f-aa25-1d6973eef5ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be60869f-1203-4f0f-aa25-1d6973eef5ef-maas-api-xks-ca\") pod \"be60869f-1203-4f0f-aa25-1d6973eef5ef\" (UID: \"be60869f-1203-4f0f-aa25-1d6973eef5ef\") "
Nov 02 13:00:21.413878 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:21.413855    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be60869f-1203-4f0f-aa25-1d6973eef5ef-maas-api-openshift-ca" pod "be60869f-1203-4f0f-aa25-1d6973eef5ef" (UID: "be60869f-1203-4f0f-aa25-1d6973eef5ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:21.415614 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:21.415589    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be60869f-1203-4f0f-aa25-1d6973eef5ef-maas-api-xks-ca" pod "be60869f-1203-4f0f-aa25-1d6973eef5ef" (UID: "be60869f-1203-4f0f-aa25-1d6973eef5ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:00:21.415614 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:21.415606    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be60869f-1203-4f0f-aa25-1d6973eef5ef-kube-api-access-t9547" pod "be60869f-1203-4f0f-aa25-1d6973eef5ef" (UID: "be60869f-1203-4f0f-aa25-1d6973eef5ef"). InnerVolumeSpecName "kube-api-access-t9547". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:21.488066 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:21.486401    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r6j5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:00:21.488066 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:21.486468    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r" podUID="f6025f4f-a2b0-4537-8a34-439e39b04658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 02 13:00:21.514882 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:21.514854    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9547\" (UniqueName: \"kubernetes.io/projected/be60869f-1203-4f0f-aa25-1d6973eef5ef-kube-api-access-t9547\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:21.514882 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:21.514879    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be60869f-1203-4f0f-aa25-1d6973eef5ef-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:21.515087 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:21.514889    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be60869f-1203-4f0f-aa25-1d6973eef5ef-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:22.194924 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:22.194887    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"
Nov 02 13:00:22.194924 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:22.194914    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457" event={"ID":"be60869f-1203-4f0f-aa25-1d6973eef5ef","Type":"ContainerDied","Data":"d7d726263b7269300fc20f04f9e501e1924b8de40996ec7fbd4a405af2c0660c"}
Nov 02 13:00:22.195163 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:22.194970    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7d726263b7269300fc20f04f9e501e1924b8de40996ec7fbd4a405af2c0660c"
Nov 02 13:00:22.484329 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:22.484286    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r6j5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:00:22.484496 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:22.484359    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r" podUID="f6025f4f-a2b0-4537-8a34-439e39b04658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 02 13:00:23.484397 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:23.484359    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r6j5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:00:23.484812 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:23.484431    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r" podUID="f6025f4f-a2b0-4537-8a34-439e39b04658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 02 13:00:24.484418 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:24.484382    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r6j5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:00:24.485040 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:24.485009    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r" podUID="f6025f4f-a2b0-4537-8a34-439e39b04658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 02 13:00:25.484352 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:25.484315    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r6j5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:00:25.484549 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:25.484392    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r" podUID="f6025f4f-a2b0-4537-8a34-439e39b04658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 02 13:00:26.484315 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:26.484273    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r6j5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:00:26.484493 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:26.484354    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r" podUID="f6025f4f-a2b0-4537-8a34-439e39b04658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 02 13:00:27.483894 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:27.483855    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r6j5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 02 13:00:27.484326 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:27.483913    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r" podUID="f6025f4f-a2b0-4537-8a34-439e39b04658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 02 13:00:28.292064 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:28.292014    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:00:29.219272 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:29.219175    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:29.219944 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:29.219909    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:29.220622 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:29.220606    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:00:30.423129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:30.423081    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-978854467-vq6wf"]
Nov 02 13:00:33.325029 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:33.324993    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:00:34.279506 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.279467    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"]
Nov 02 13:00:34.280080 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.280045    4464 cpu_manager.go:414] "containerMap: removing container" podUID="be60869f-1203-4f0f-aa25-1d6973eef5ef" containerName="revoke-keys"
Nov 02 13:00:34.280080 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.280073    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be60869f-1203-4f0f-aa25-1d6973eef5ef" containerName="revoke-keys"
Nov 02 13:00:34.280307 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.280150    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="be60869f-1203-4f0f-aa25-1d6973eef5ef" containerName="revoke-keys"
Nov 02 13:00:34.283641 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.283614    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.345230 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.345199    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.345653 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.345253    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.345653 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.345271    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.345653 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.345405    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.345653 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.345436    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.345653 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.345478    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.345653 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.345566    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.345653 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.345607    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.345899 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.345692    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g5ft\" (UniqueName: \"kubernetes.io/projected/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-kube-api-access-5g5ft\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.389774 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.389747    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"]
Nov 02 13:00:34.447107 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.447078    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.447253 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.447125    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.447253 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.447148    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.447253 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.447195    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.447253 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.447226    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.447484 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.447254    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.447484 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.447287    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g5ft\" (UniqueName: \"kubernetes.io/projected/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-kube-api-access-5g5ft\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.447484 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.447380    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.447484 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.447448    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.447715 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.447603    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.447715 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.447665    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.448075 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.448045    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.448204 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.448148    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.448570 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.448541    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.449577 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.449557    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.449663 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.449603    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.509333 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.509309    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g5ft\" (UniqueName: \"kubernetes.io/projected/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-kube-api-access-5g5ft\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.509418 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.509330    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-wktxt\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:34.596569 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:34.596498    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:35.023096 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:35.023062    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"]
Nov 02 13:00:35.024369 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:00:35.024342    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb595a6ad_3c8e_4a19_b22e_20b5b3fe2b51.slice/crio-8838b76271bcdc0dbd15c53c8c93c369785e28b34028d669b1391ff039773b32 WatchSource:0}: Error finding container 8838b76271bcdc0dbd15c53c8c93c369785e28b34028d669b1391ff039773b32: Status 404 returned error can't find the container with id 8838b76271bcdc0dbd15c53c8c93c369785e28b34028d669b1391ff039773b32
Nov 02 13:00:35.026601 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:35.026570    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:00:35.026696 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:35.026634    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:00:35.026696 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:35.026669    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:00:35.268960 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:35.268920    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt" event={"ID":"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51","Type":"ContainerStarted","Data":"0d4a1aa1e9cd5b3562ad3d58571f31758fa096a7febb0c84e16327056d247e38"}
Nov 02 13:00:35.269184 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:35.269011    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt" event={"ID":"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51","Type":"ContainerStarted","Data":"8838b76271bcdc0dbd15c53c8c93c369785e28b34028d669b1391ff039773b32"}
Nov 02 13:00:35.394822 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:35.394719    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt" podStartSLOduration=1.3947057489999999 podStartE2EDuration="1.394705749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:35.392010734 +0000 UTC m=+3863.955719144" watchObservedRunningTime="2026-11-02 13:00:35.394705749 +0000 UTC m=+3863.958414158"
Nov 02 13:00:35.424304 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:35.424252    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf" podUID="53a6e480-9610-45ac-9e6a-3b5658f56f6c" containerName="payload-processing" containerID="cri-o://de8f95e850215850da909a029a04696107e6eb2786e7f89bfc6fd2d7d51f21fe" gracePeriod=25
Nov 02 13:00:35.602003 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:35.598076    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:35.806869 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:35.806831    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 13:00:36.006047 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:35.997266    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53a6e480-9610-45ac-9e6a-3b5658f56f6c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53a6e480-9610-45ac-9e6a-3b5658f56f6c-routing-overlay-volume\") pod \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\" (UID: \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\") "
Nov 02 13:00:36.006047 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:35.997356    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53a6e480-9610-45ac-9e6a-3b5658f56f6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a6e480-9610-45ac-9e6a-3b5658f56f6c-plugins-config-volume\") pod \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\" (UID: \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\") "
Nov 02 13:00:36.006047 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:35.997483    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a6e480-9610-45ac-9e6a-3b5658f56f6c-kube-api-access-69z6p\" (UniqueName: \"kubernetes.io/projected/53a6e480-9610-45ac-9e6a-3b5658f56f6c-kube-api-access-69z6p\") pod \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\" (UID: \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\") "
Nov 02 13:00:36.006047 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:35.997599    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a6e480-9610-45ac-9e6a-3b5658f56f6c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/53a6e480-9610-45ac-9e6a-3b5658f56f6c-provider-credentials\") pod \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\" (UID: \"53a6e480-9610-45ac-9e6a-3b5658f56f6c\") "
Nov 02 13:00:36.006047 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.003659    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a6e480-9610-45ac-9e6a-3b5658f56f6c-plugins-config-volume" pod "53a6e480-9610-45ac-9e6a-3b5658f56f6c" (UID: "53a6e480-9610-45ac-9e6a-3b5658f56f6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:36.006047 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.004060    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a6e480-9610-45ac-9e6a-3b5658f56f6c-routing-overlay-volume" pod "53a6e480-9610-45ac-9e6a-3b5658f56f6c" (UID: "53a6e480-9610-45ac-9e6a-3b5658f56f6c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:36.006047 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.005915    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a6e480-9610-45ac-9e6a-3b5658f56f6c-provider-credentials" pod "53a6e480-9610-45ac-9e6a-3b5658f56f6c" (UID: "53a6e480-9610-45ac-9e6a-3b5658f56f6c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:36.008082 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.008052    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a6e480-9610-45ac-9e6a-3b5658f56f6c-kube-api-access-69z6p" pod "53a6e480-9610-45ac-9e6a-3b5658f56f6c" (UID: "53a6e480-9610-45ac-9e6a-3b5658f56f6c"). InnerVolumeSpecName "kube-api-access-69z6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:36.102722 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.102672    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a6e480-9610-45ac-9e6a-3b5658f56f6c-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:36.102722 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.102716    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69z6p\" (UniqueName: \"kubernetes.io/projected/53a6e480-9610-45ac-9e6a-3b5658f56f6c-kube-api-access-69z6p\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:36.103051 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.102733    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53a6e480-9610-45ac-9e6a-3b5658f56f6c-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:36.103051 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.102751    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53a6e480-9610-45ac-9e6a-3b5658f56f6c-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:36.299090 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.298436    4464 generic.go:356] "Generic (PLEG): container finished" podID="53a6e480-9610-45ac-9e6a-3b5658f56f6c" containerID="de8f95e850215850da909a029a04696107e6eb2786e7f89bfc6fd2d7d51f21fe" exitCode=0
Nov 02 13:00:36.299090 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.298514    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf"
Nov 02 13:00:36.299090 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.298844    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf" event={"ID":"53a6e480-9610-45ac-9e6a-3b5658f56f6c","Type":"ContainerDied","Data":"de8f95e850215850da909a029a04696107e6eb2786e7f89bfc6fd2d7d51f21fe"}
Nov 02 13:00:36.299090 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.298902    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-978854467-vq6wf" event={"ID":"53a6e480-9610-45ac-9e6a-3b5658f56f6c","Type":"ContainerDied","Data":"3cb25768d20af77a1667481e9998dcbd95b0f8ca4f3389f36e54ad10af1efac7"}
Nov 02 13:00:36.299090 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.298939    4464 scope.go:122] "RemoveContainer" containerID="de8f95e850215850da909a029a04696107e6eb2786e7f89bfc6fd2d7d51f21fe"
Nov 02 13:00:36.311906 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.311884    4464 scope.go:122] "RemoveContainer" containerID="de8f95e850215850da909a029a04696107e6eb2786e7f89bfc6fd2d7d51f21fe"
Nov 02 13:00:36.312234 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:36.312215    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8f95e850215850da909a029a04696107e6eb2786e7f89bfc6fd2d7d51f21fe\": container with ID starting with de8f95e850215850da909a029a04696107e6eb2786e7f89bfc6fd2d7d51f21fe not found: ID does not exist" containerID="de8f95e850215850da909a029a04696107e6eb2786e7f89bfc6fd2d7d51f21fe"
Nov 02 13:00:36.312318 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.312244    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8f95e850215850da909a029a04696107e6eb2786e7f89bfc6fd2d7d51f21fe"} err="failed to get container status \"de8f95e850215850da909a029a04696107e6eb2786e7f89bfc6fd2d7d51f21fe\": rpc error: code = NotFound desc = could not find container \"de8f95e850215850da909a029a04696107e6eb2786e7f89bfc6fd2d7d51f21fe\": container with ID starting with de8f95e850215850da909a029a04696107e6eb2786e7f89bfc6fd2d7d51f21fe not found: ID does not exist"
Nov 02 13:00:36.497002 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.490228    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-978854467-vq6wf"]
Nov 02 13:00:36.518076 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.518041    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-978854467-vq6wf"]
Nov 02 13:00:36.599612 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.599221    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-wktxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Nov 02 13:00:36.599612 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:36.599290    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt" podUID="b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Nov 02 13:00:37.589773 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:37.589743    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53a6e480-9610-45ac-9e6a-3b5658f56f6c" path="/var/lib/kubelet/pods/53a6e480-9610-45ac-9e6a-3b5658f56f6c/volumes"
Nov 02 13:00:37.597792 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:37.597771    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-wktxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Nov 02 13:00:37.597868 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:37.597815    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt" podUID="b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Nov 02 13:00:38.017564 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.017529    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"]
Nov 02 13:00:38.018331 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.018312    4464 cpu_manager.go:414] "containerMap: removing container" podUID="53a6e480-9610-45ac-9e6a-3b5658f56f6c" containerName="payload-processing"
Nov 02 13:00:38.018381 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.018334    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a6e480-9610-45ac-9e6a-3b5658f56f6c" containerName="payload-processing"
Nov 02 13:00:38.018454 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.018442    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a6e480-9610-45ac-9e6a-3b5658f56f6c" containerName="payload-processing"
Nov 02 13:00:38.023295 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.023276    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"
Nov 02 13:00:38.088910 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.088530    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"]
Nov 02 13:00:38.123569 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.123539    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nhqp\" (UniqueName: \"kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-kube-api-access-4nhqp\") pod \"payload-processing-external-model-599b76764c-tgzgs\" (UID: \"36b4431c-8123-44bc-826e-d4f440aede12\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"
Nov 02 13:00:38.123726 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.123662    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b4431c-8123-44bc-826e-d4f440aede12-plugins-config-volume\") pod \"payload-processing-external-model-599b76764c-tgzgs\" (UID: \"36b4431c-8123-44bc-826e-d4f440aede12\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"
Nov 02 13:00:38.123792 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.123750    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36b4431c-8123-44bc-826e-d4f440aede12-routing-overlay-volume\") pod \"payload-processing-external-model-599b76764c-tgzgs\" (UID: \"36b4431c-8123-44bc-826e-d4f440aede12\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"
Nov 02 13:00:38.123841 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.123789    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-provider-credentials\") pod \"payload-processing-external-model-599b76764c-tgzgs\" (UID: \"36b4431c-8123-44bc-826e-d4f440aede12\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"
Nov 02 13:00:38.136060 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.136035    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 13:00:38.136060 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.136051    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-13ffe4f19df1\""
Nov 02 13:00:38.136060 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.136051    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 13:00:38.136266 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.136040    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 13:00:38.136266 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.136079    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 13:00:38.136266 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.136122    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-gxk98\""
Nov 02 13:00:38.224579 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.224548    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nhqp\" (UniqueName: \"kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-kube-api-access-4nhqp\") pod \"payload-processing-external-model-599b76764c-tgzgs\" (UID: \"36b4431c-8123-44bc-826e-d4f440aede12\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"
Nov 02 13:00:38.224740 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.224632    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b4431c-8123-44bc-826e-d4f440aede12-plugins-config-volume\") pod \"payload-processing-external-model-599b76764c-tgzgs\" (UID: \"36b4431c-8123-44bc-826e-d4f440aede12\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"
Nov 02 13:00:38.224740 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.224699    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36b4431c-8123-44bc-826e-d4f440aede12-routing-overlay-volume\") pod \"payload-processing-external-model-599b76764c-tgzgs\" (UID: \"36b4431c-8123-44bc-826e-d4f440aede12\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"
Nov 02 13:00:38.224871 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.224743    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-provider-credentials\") pod \"payload-processing-external-model-599b76764c-tgzgs\" (UID: \"36b4431c-8123-44bc-826e-d4f440aede12\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"
Nov 02 13:00:38.225378 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.225353    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36b4431c-8123-44bc-826e-d4f440aede12-routing-overlay-volume\") pod \"payload-processing-external-model-599b76764c-tgzgs\" (UID: \"36b4431c-8123-44bc-826e-d4f440aede12\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"
Nov 02 13:00:38.225470 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.225389    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b4431c-8123-44bc-826e-d4f440aede12-plugins-config-volume\") pod \"payload-processing-external-model-599b76764c-tgzgs\" (UID: \"36b4431c-8123-44bc-826e-d4f440aede12\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"
Nov 02 13:00:38.227114 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.227094    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-provider-credentials\") pod \"payload-processing-external-model-599b76764c-tgzgs\" (UID: \"36b4431c-8123-44bc-826e-d4f440aede12\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"
Nov 02 13:00:38.287280 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.287216    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nhqp\" (UniqueName: \"kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-kube-api-access-4nhqp\") pod \"payload-processing-external-model-599b76764c-tgzgs\" (UID: \"36b4431c-8123-44bc-826e-d4f440aede12\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"
Nov 02 13:00:38.338274 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.338253    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"
Nov 02 13:00:38.597087 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.597008    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-wktxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Nov 02 13:00:38.597087 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.597058    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt" podUID="b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Nov 02 13:00:38.748810 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:38.748774    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"]
Nov 02 13:00:38.749814 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:00:38.749787    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b4431c_8123_44bc_826e_d4f440aede12.slice/crio-40bdd5ee2efe2db0f17d737c82f60905d8f0b9fc38e4ebe300e07252446471f2 WatchSource:0}: Error finding container 40bdd5ee2efe2db0f17d737c82f60905d8f0b9fc38e4ebe300e07252446471f2: Status 404 returned error can't find the container with id 40bdd5ee2efe2db0f17d737c82f60905d8f0b9fc38e4ebe300e07252446471f2
Nov 02 13:00:39.320299 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:39.320264    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs" event={"ID":"36b4431c-8123-44bc-826e-d4f440aede12","Type":"ContainerStarted","Data":"c2af2c8f89cb4785e57a734ae68ff2916f9f415c87c54f6c7fba85ae782959b0"}
Nov 02 13:00:39.320486 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:39.320321    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs" event={"ID":"36b4431c-8123-44bc-826e-d4f440aede12","Type":"ContainerStarted","Data":"40bdd5ee2efe2db0f17d737c82f60905d8f0b9fc38e4ebe300e07252446471f2"}
Nov 02 13:00:39.320486 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:39.320344    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"
Nov 02 13:00:39.336653 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:39.336614    4464 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-13ffe4f19df1: secret "agc-cred-13ffe4f19df1" not found
Nov 02 13:00:39.336653 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:39.336638    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-599b76764c-tgzgs: secret "agc-cred-13ffe4f19df1" not found
Nov 02 13:00:39.336883 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:39.336703    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-provider-credentials podName:36b4431c-8123-44bc-826e-d4f440aede12 nodeName:}" failed. No retries permitted until 2026-11-02 13:00:39.836683855 +0000 UTC m=+3868.400392263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-provider-credentials") pod "payload-processing-external-model-599b76764c-tgzgs" (UID: "36b4431c-8123-44bc-826e-d4f440aede12") : secret "agc-cred-13ffe4f19df1" not found
Nov 02 13:00:39.443175 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:39.443114    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs" podStartSLOduration=2.443100378 podStartE2EDuration="2.443100378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:39.442710549 +0000 UTC m=+3868.006418977" watchObservedRunningTime="2026-11-02 13:00:39.443100378 +0000 UTC m=+3868.006808797"
Nov 02 13:00:39.598143 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:39.598061    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-wktxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Nov 02 13:00:39.598143 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:39.598120    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt" podUID="b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Nov 02 13:00:39.841352 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:39.841320    4464 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-13ffe4f19df1: secret "agc-cred-13ffe4f19df1" not found
Nov 02 13:00:39.841352 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:39.841347    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-599b76764c-tgzgs: secret "agc-cred-13ffe4f19df1" not found
Nov 02 13:00:39.841600 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:39.841403    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-provider-credentials podName:36b4431c-8123-44bc-826e-d4f440aede12 nodeName:}" failed. No retries permitted until 2026-11-02 13:00:40.841387437 +0000 UTC m=+3869.405095823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-provider-credentials") pod "payload-processing-external-model-599b76764c-tgzgs" (UID: "36b4431c-8123-44bc-826e-d4f440aede12") : secret "agc-cred-13ffe4f19df1" not found
Nov 02 13:00:40.596892 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:40.596849    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-wktxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Nov 02 13:00:40.597102 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:40.596909    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt" podUID="b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Nov 02 13:00:40.851845 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:40.851770    4464 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-13ffe4f19df1: secret "agc-cred-13ffe4f19df1" not found
Nov 02 13:00:40.851845 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:40.851796    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-599b76764c-tgzgs: secret "agc-cred-13ffe4f19df1" not found
Nov 02 13:00:40.851845 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:40.851845    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-provider-credentials podName:36b4431c-8123-44bc-826e-d4f440aede12 nodeName:}" failed. No retries permitted until 2026-11-02 13:00:42.851831591 +0000 UTC m=+3871.415539996 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-provider-credentials") pod "payload-processing-external-model-599b76764c-tgzgs" (UID: "36b4431c-8123-44bc-826e-d4f440aede12") : secret "agc-cred-13ffe4f19df1" not found
Nov 02 13:00:40.936653 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:40.936617    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"]
Nov 02 13:00:41.597604 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:41.597572    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-wktxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Nov 02 13:00:41.597780 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:41.597622    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt" podUID="b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Nov 02 13:00:42.597681 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:42.597647    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-wktxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Nov 02 13:00:42.598143 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:42.597701    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt" podUID="b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Nov 02 13:00:42.873159 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:42.873084    4464 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-13ffe4f19df1: secret "agc-cred-13ffe4f19df1" not found
Nov 02 13:00:42.873159 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:42.873111    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-599b76764c-tgzgs: secret "agc-cred-13ffe4f19df1" not found
Nov 02 13:00:42.873329 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:42.873164    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-provider-credentials podName:36b4431c-8123-44bc-826e-d4f440aede12 nodeName:}" failed. No retries permitted until 2026-11-02 13:00:46.873148048 +0000 UTC m=+3875.436856445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-provider-credentials") pod "payload-processing-external-model-599b76764c-tgzgs" (UID: "36b4431c-8123-44bc-826e-d4f440aede12") : secret "agc-cred-13ffe4f19df1" not found
Nov 02 13:00:44.460391 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:44.460357    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:44.460901 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:44.460835    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:44.461682 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:44.461661    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:00:46.330845 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:46.330802    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs" podUID="36b4431c-8123-44bc-826e-d4f440aede12" containerName="payload-processing" containerID="cri-o://c2af2c8f89cb4785e57a734ae68ff2916f9f415c87c54f6c7fba85ae782959b0" gracePeriod=25
Nov 02 13:00:46.594178 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:46.594117    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"
Nov 02 13:00:46.717412 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:46.717373    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-kube-api-access-4nhqp\" (UniqueName: \"kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-kube-api-access-4nhqp\") pod \"36b4431c-8123-44bc-826e-d4f440aede12\" (UID: \"36b4431c-8123-44bc-826e-d4f440aede12\") "
Nov 02 13:00:46.717627 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:46.717489    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b4431c-8123-44bc-826e-d4f440aede12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b4431c-8123-44bc-826e-d4f440aede12-plugins-config-volume\") pod \"36b4431c-8123-44bc-826e-d4f440aede12\" (UID: \"36b4431c-8123-44bc-826e-d4f440aede12\") "
Nov 02 13:00:46.718564 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:46.718519    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b4431c-8123-44bc-826e-d4f440aede12-plugins-config-volume" pod "36b4431c-8123-44bc-826e-d4f440aede12" (UID: "36b4431c-8123-44bc-826e-d4f440aede12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:46.720516 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:46.720138    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-provider-credentials\" (UniqueName: \"kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-provider-credentials\") pod \"36b4431c-8123-44bc-826e-d4f440aede12\" (UID: \"36b4431c-8123-44bc-826e-d4f440aede12\") "
Nov 02 13:00:46.726029 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:46.720692    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b4431c-8123-44bc-826e-d4f440aede12-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36b4431c-8123-44bc-826e-d4f440aede12-routing-overlay-volume\") pod \"36b4431c-8123-44bc-826e-d4f440aede12\" (UID: \"36b4431c-8123-44bc-826e-d4f440aede12\") "
Nov 02 13:00:46.726029 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:46.721087    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b4431c-8123-44bc-826e-d4f440aede12-routing-overlay-volume" pod "36b4431c-8123-44bc-826e-d4f440aede12" (UID: "36b4431c-8123-44bc-826e-d4f440aede12"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:46.726029 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:46.723706    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-kube-api-access-4nhqp" pod "36b4431c-8123-44bc-826e-d4f440aede12" (UID: "36b4431c-8123-44bc-826e-d4f440aede12"). InnerVolumeSpecName "kube-api-access-4nhqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:46.726029 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:46.725405    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nhqp\" (UniqueName: \"kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-kube-api-access-4nhqp\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:46.726029 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:46.725431    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b4431c-8123-44bc-826e-d4f440aede12-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:46.726029 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:46.725449    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36b4431c-8123-44bc-826e-d4f440aede12-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:46.726652 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:46.726619    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-provider-credentials" pod "36b4431c-8123-44bc-826e-d4f440aede12" (UID: "36b4431c-8123-44bc-826e-d4f440aede12"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:46.826650 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:46.826604    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36b4431c-8123-44bc-826e-d4f440aede12-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:47.380588 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:47.380550    4464 generic.go:356] "Generic (PLEG): container finished" podID="36b4431c-8123-44bc-826e-d4f440aede12" containerID="c2af2c8f89cb4785e57a734ae68ff2916f9f415c87c54f6c7fba85ae782959b0" exitCode=0
Nov 02 13:00:47.384207 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:47.380710    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs" event={"ID":"36b4431c-8123-44bc-826e-d4f440aede12","Type":"ContainerDied","Data":"c2af2c8f89cb4785e57a734ae68ff2916f9f415c87c54f6c7fba85ae782959b0"}
Nov 02 13:00:47.384207 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:47.380789    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs" event={"ID":"36b4431c-8123-44bc-826e-d4f440aede12","Type":"ContainerDied","Data":"40bdd5ee2efe2db0f17d737c82f60905d8f0b9fc38e4ebe300e07252446471f2"}
Nov 02 13:00:47.384207 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:47.380823    4464 scope.go:122] "RemoveContainer" containerID="c2af2c8f89cb4785e57a734ae68ff2916f9f415c87c54f6c7fba85ae782959b0"
Nov 02 13:00:47.384207 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:47.381039    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"
Nov 02 13:00:47.400056 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:47.400032    4464 scope.go:122] "RemoveContainer" containerID="c2af2c8f89cb4785e57a734ae68ff2916f9f415c87c54f6c7fba85ae782959b0"
Nov 02 13:00:47.400375 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:47.400354    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2af2c8f89cb4785e57a734ae68ff2916f9f415c87c54f6c7fba85ae782959b0\": container with ID starting with c2af2c8f89cb4785e57a734ae68ff2916f9f415c87c54f6c7fba85ae782959b0 not found: ID does not exist" containerID="c2af2c8f89cb4785e57a734ae68ff2916f9f415c87c54f6c7fba85ae782959b0"
Nov 02 13:00:47.400464 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:47.400382    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2af2c8f89cb4785e57a734ae68ff2916f9f415c87c54f6c7fba85ae782959b0"} err="failed to get container status \"c2af2c8f89cb4785e57a734ae68ff2916f9f415c87c54f6c7fba85ae782959b0\": rpc error: code = NotFound desc = could not find container \"c2af2c8f89cb4785e57a734ae68ff2916f9f415c87c54f6c7fba85ae782959b0\": container with ID starting with c2af2c8f89cb4785e57a734ae68ff2916f9f415c87c54f6c7fba85ae782959b0 not found: ID does not exist"
Nov 02 13:00:47.549660 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:47.549626    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"]
Nov 02 13:00:47.555916 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:47.555886    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-599b76764c-tgzgs"]
Nov 02 13:00:47.580699 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:47.580670    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b4431c-8123-44bc-826e-d4f440aede12" path="/var/lib/kubelet/pods/36b4431c-8123-44bc-826e-d4f440aede12/volumes"
Nov 02 13:00:48.464145 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.464105    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"]
Nov 02 13:00:48.464808 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.464785    4464 cpu_manager.go:414] "containerMap: removing container" podUID="36b4431c-8123-44bc-826e-d4f440aede12" containerName="payload-processing"
Nov 02 13:00:48.464909 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.464809    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b4431c-8123-44bc-826e-d4f440aede12" containerName="payload-processing"
Nov 02 13:00:48.465006 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.464960    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b4431c-8123-44bc-826e-d4f440aede12" containerName="payload-processing"
Nov 02 13:00:48.469490 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.469468    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"
Nov 02 13:00:48.523657 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.523637    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-13ffe4f19df1\""
Nov 02 13:00:48.526990 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.526957    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"]
Nov 02 13:00:48.543805 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.543775    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn4km\" (UniqueName: \"kubernetes.io/projected/1e3f4640-9283-4368-87df-553a219a0641-kube-api-access-sn4km\") pod \"payload-processing-external-model-599b76764c-9xnbt\" (UID: \"1e3f4640-9283-4368-87df-553a219a0641\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"
Nov 02 13:00:48.543911 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.543819    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e3f4640-9283-4368-87df-553a219a0641-routing-overlay-volume\") pod \"payload-processing-external-model-599b76764c-9xnbt\" (UID: \"1e3f4640-9283-4368-87df-553a219a0641\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"
Nov 02 13:00:48.543985 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.543930    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e3f4640-9283-4368-87df-553a219a0641-provider-credentials\") pod \"payload-processing-external-model-599b76764c-9xnbt\" (UID: \"1e3f4640-9283-4368-87df-553a219a0641\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"
Nov 02 13:00:48.544053 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.544029    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3f4640-9283-4368-87df-553a219a0641-plugins-config-volume\") pod \"payload-processing-external-model-599b76764c-9xnbt\" (UID: \"1e3f4640-9283-4368-87df-553a219a0641\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"
Nov 02 13:00:48.574744 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.574717    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 13:00:48.574887 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.574771    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 13:00:48.574887 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.574791    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 13:00:48.574887 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.574791    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 13:00:48.623822 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.623801    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-p9f94\""
Nov 02 13:00:48.645404 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.645380    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn4km\" (UniqueName: \"kubernetes.io/projected/1e3f4640-9283-4368-87df-553a219a0641-kube-api-access-sn4km\") pod \"payload-processing-external-model-599b76764c-9xnbt\" (UID: \"1e3f4640-9283-4368-87df-553a219a0641\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"
Nov 02 13:00:48.645510 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.645413    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e3f4640-9283-4368-87df-553a219a0641-routing-overlay-volume\") pod \"payload-processing-external-model-599b76764c-9xnbt\" (UID: \"1e3f4640-9283-4368-87df-553a219a0641\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"
Nov 02 13:00:48.645510 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.645459    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e3f4640-9283-4368-87df-553a219a0641-provider-credentials\") pod \"payload-processing-external-model-599b76764c-9xnbt\" (UID: \"1e3f4640-9283-4368-87df-553a219a0641\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"
Nov 02 13:00:48.645510 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.645499    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3f4640-9283-4368-87df-553a219a0641-plugins-config-volume\") pod \"payload-processing-external-model-599b76764c-9xnbt\" (UID: \"1e3f4640-9283-4368-87df-553a219a0641\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"
Nov 02 13:00:48.646246 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.646225    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e3f4640-9283-4368-87df-553a219a0641-routing-overlay-volume\") pod \"payload-processing-external-model-599b76764c-9xnbt\" (UID: \"1e3f4640-9283-4368-87df-553a219a0641\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"
Nov 02 13:00:48.646348 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.646332    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3f4640-9283-4368-87df-553a219a0641-plugins-config-volume\") pod \"payload-processing-external-model-599b76764c-9xnbt\" (UID: \"1e3f4640-9283-4368-87df-553a219a0641\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"
Nov 02 13:00:48.647949 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.647928    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e3f4640-9283-4368-87df-553a219a0641-provider-credentials\") pod \"payload-processing-external-model-599b76764c-9xnbt\" (UID: \"1e3f4640-9283-4368-87df-553a219a0641\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"
Nov 02 13:00:48.707284 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.707259    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn4km\" (UniqueName: \"kubernetes.io/projected/1e3f4640-9283-4368-87df-553a219a0641-kube-api-access-sn4km\") pod \"payload-processing-external-model-599b76764c-9xnbt\" (UID: \"1e3f4640-9283-4368-87df-553a219a0641\") " pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"
Nov 02 13:00:48.782240 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:48.782212    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"
Nov 02 13:00:49.157589 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:49.157565    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"]
Nov 02 13:00:49.158817 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:00:49.158788    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3f4640_9283_4368_87df_553a219a0641.slice/crio-55938e8078c443659701ba2f25087ca370ba59322ca165401a6e50b2ffba98a5 WatchSource:0}: Error finding container 55938e8078c443659701ba2f25087ca370ba59322ca165401a6e50b2ffba98a5: Status 404 returned error can't find the container with id 55938e8078c443659701ba2f25087ca370ba59322ca165401a6e50b2ffba98a5
Nov 02 13:00:49.393570 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:49.393533    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt" event={"ID":"1e3f4640-9283-4368-87df-553a219a0641","Type":"ContainerStarted","Data":"44e30f3607669f93b726e7e5f94d60d7ceb7a8e8f5fc6521d2bda23df9aad246"}
Nov 02 13:00:49.393745 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:49.393595    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt" event={"ID":"1e3f4640-9283-4368-87df-553a219a0641","Type":"ContainerStarted","Data":"55938e8078c443659701ba2f25087ca370ba59322ca165401a6e50b2ffba98a5"}
Nov 02 13:00:49.393745 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:49.393624    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"
Nov 02 13:00:49.514457 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:49.514408    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt" podStartSLOduration=1.514396247 podStartE2EDuration="1.514396247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:49.512464913 +0000 UTC m=+3878.076173325" watchObservedRunningTime="2026-11-02 13:00:49.514396247 +0000 UTC m=+3878.078104657"
Nov 02 13:00:52.094015 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:52.093964    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"]
Nov 02 13:00:57.095104 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.095037    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt" podUID="1e3f4640-9283-4368-87df-553a219a0641" containerName="payload-processing" containerID="cri-o://44e30f3607669f93b726e7e5f94d60d7ceb7a8e8f5fc6521d2bda23df9aad246" gracePeriod=25
Nov 02 13:00:57.351303 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.351241    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"
Nov 02 13:00:57.437196 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.437154    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3f4640-9283-4368-87df-553a219a0641-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3f4640-9283-4368-87df-553a219a0641-plugins-config-volume\") pod \"1e3f4640-9283-4368-87df-553a219a0641\" (UID: \"1e3f4640-9283-4368-87df-553a219a0641\") "
Nov 02 13:00:57.437396 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.437236    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3f4640-9283-4368-87df-553a219a0641-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e3f4640-9283-4368-87df-553a219a0641-provider-credentials\") pod \"1e3f4640-9283-4368-87df-553a219a0641\" (UID: \"1e3f4640-9283-4368-87df-553a219a0641\") "
Nov 02 13:00:57.437396 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.437359    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e3f4640-9283-4368-87df-553a219a0641-kube-api-access-sn4km\" (UniqueName: \"kubernetes.io/projected/1e3f4640-9283-4368-87df-553a219a0641-kube-api-access-sn4km\") pod \"1e3f4640-9283-4368-87df-553a219a0641\" (UID: \"1e3f4640-9283-4368-87df-553a219a0641\") "
Nov 02 13:00:57.437396 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.437389    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e3f4640-9283-4368-87df-553a219a0641-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e3f4640-9283-4368-87df-553a219a0641-routing-overlay-volume\") pod \"1e3f4640-9283-4368-87df-553a219a0641\" (UID: \"1e3f4640-9283-4368-87df-553a219a0641\") "
Nov 02 13:00:57.437639 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.437612    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3f4640-9283-4368-87df-553a219a0641-plugins-config-volume" pod "1e3f4640-9283-4368-87df-553a219a0641" (UID: "1e3f4640-9283-4368-87df-553a219a0641"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:57.437739 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.437718    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e3f4640-9283-4368-87df-553a219a0641-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:57.438000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.437964    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e3f4640-9283-4368-87df-553a219a0641-routing-overlay-volume" pod "1e3f4640-9283-4368-87df-553a219a0641" (UID: "1e3f4640-9283-4368-87df-553a219a0641"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:57.438770 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.438747    4464 generic.go:356] "Generic (PLEG): container finished" podID="1e3f4640-9283-4368-87df-553a219a0641" containerID="44e30f3607669f93b726e7e5f94d60d7ceb7a8e8f5fc6521d2bda23df9aad246" exitCode=0
Nov 02 13:00:57.438877 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.438793    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"
Nov 02 13:00:57.438877 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.438839    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt" event={"ID":"1e3f4640-9283-4368-87df-553a219a0641","Type":"ContainerDied","Data":"44e30f3607669f93b726e7e5f94d60d7ceb7a8e8f5fc6521d2bda23df9aad246"}
Nov 02 13:00:57.438971 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.438910    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-599b76764c-9xnbt" event={"ID":"1e3f4640-9283-4368-87df-553a219a0641","Type":"ContainerDied","Data":"55938e8078c443659701ba2f25087ca370ba59322ca165401a6e50b2ffba98a5"}
Nov 02 13:00:57.438971 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.438946    4464 scope.go:122] "RemoveContainer" containerID="44e30f3607669f93b726e7e5f94d60d7ceb7a8e8f5fc6521d2bda23df9aad246"
Nov 02 13:00:57.439537 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.439508    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3f4640-9283-4368-87df-553a219a0641-provider-credentials" pod "1e3f4640-9283-4368-87df-553a219a0641" (UID: "1e3f4640-9283-4368-87df-553a219a0641"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:57.439632 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.439571    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3f4640-9283-4368-87df-553a219a0641-kube-api-access-sn4km" pod "1e3f4640-9283-4368-87df-553a219a0641" (UID: "1e3f4640-9283-4368-87df-553a219a0641"). InnerVolumeSpecName "kube-api-access-sn4km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:57.460141 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.459635    4464 scope.go:122] "RemoveContainer" containerID="44e30f3607669f93b726e7e5f94d60d7ceb7a8e8f5fc6521d2bda23df9aad246"
Nov 02 13:00:57.460223 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:57.460153    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e30f3607669f93b726e7e5f94d60d7ceb7a8e8f5fc6521d2bda23df9aad246\": container with ID starting with 44e30f3607669f93b726e7e5f94d60d7ceb7a8e8f5fc6521d2bda23df9aad246 not found: ID does not exist" containerID="44e30f3607669f93b726e7e5f94d60d7ceb7a8e8f5fc6521d2bda23df9aad246"
Nov 02 13:00:57.460223 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.460189    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e30f3607669f93b726e7e5f94d60d7ceb7a8e8f5fc6521d2bda23df9aad246"} err="failed to get container status \"44e30f3607669f93b726e7e5f94d60d7ceb7a8e8f5fc6521d2bda23df9aad246\": rpc error: code = NotFound desc = could not find container \"44e30f3607669f93b726e7e5f94d60d7ceb7a8e8f5fc6521d2bda23df9aad246\": container with ID starting with 44e30f3607669f93b726e7e5f94d60d7ceb7a8e8f5fc6521d2bda23df9aad246 not found: ID does not exist"
Nov 02 13:00:57.538798 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.538761    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn4km\" (UniqueName: \"kubernetes.io/projected/1e3f4640-9283-4368-87df-553a219a0641-kube-api-access-sn4km\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:57.538798 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.538797    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e3f4640-9283-4368-87df-553a219a0641-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:57.538798 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.538812    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e3f4640-9283-4368-87df-553a219a0641-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:00:57.855515 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.855482    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"]
Nov 02 13:00:57.906344 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:57.906308    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-599b76764c-9xnbt"]
Nov 02 13:00:58.292131 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:00:58.292096    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:00:59.581613 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.581579    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e3f4640-9283-4368-87df-553a219a0641" path="/var/lib/kubelet/pods/1e3f4640-9283-4368-87df-553a219a0641/volumes"
Nov 02 13:00:59.655751 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.655714    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"]
Nov 02 13:00:59.656313 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.656292    4464 cpu_manager.go:414] "containerMap: removing container" podUID="1e3f4640-9283-4368-87df-553a219a0641" containerName="payload-processing"
Nov 02 13:00:59.656313 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.656313    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e3f4640-9283-4368-87df-553a219a0641" containerName="payload-processing"
Nov 02 13:00:59.656481 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.656419    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e3f4640-9283-4368-87df-553a219a0641" containerName="payload-processing"
Nov 02 13:00:59.661184 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.661162    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"
Nov 02 13:00:59.715189 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.715162    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-897c4765e686\""
Nov 02 13:00:59.717751 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.717727    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"]
Nov 02 13:00:59.765662 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.765634    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 13:00:59.765831 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.765711    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 13:00:59.765831 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.765743    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 13:00:59.765831 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.765715    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 13:00:59.765831 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.765770    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vv95l\""
Nov 02 13:00:59.766105 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.766088    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-plugins-config-volume\") pod \"payload-processing-external-model-59f9b4bfb5-ww55q\" (UID: \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\") " pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"
Nov 02 13:00:59.766254 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.766235    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-provider-credentials\") pod \"payload-processing-external-model-59f9b4bfb5-ww55q\" (UID: \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\") " pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"
Nov 02 13:00:59.766297 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.766288    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-958pn\" (UniqueName: \"kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-kube-api-access-958pn\") pod \"payload-processing-external-model-59f9b4bfb5-ww55q\" (UID: \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\") " pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"
Nov 02 13:00:59.766340 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.766324    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-routing-overlay-volume\") pod \"payload-processing-external-model-59f9b4bfb5-ww55q\" (UID: \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\") " pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"
Nov 02 13:00:59.867242 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.867163    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-provider-credentials\") pod \"payload-processing-external-model-59f9b4bfb5-ww55q\" (UID: \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\") " pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"
Nov 02 13:00:59.867242 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.867203    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-958pn\" (UniqueName: \"kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-kube-api-access-958pn\") pod \"payload-processing-external-model-59f9b4bfb5-ww55q\" (UID: \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\") " pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"
Nov 02 13:00:59.867452 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.867242    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-routing-overlay-volume\") pod \"payload-processing-external-model-59f9b4bfb5-ww55q\" (UID: \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\") " pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"
Nov 02 13:00:59.867452 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.867295    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-plugins-config-volume\") pod \"payload-processing-external-model-59f9b4bfb5-ww55q\" (UID: \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\") " pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"
Nov 02 13:00:59.868053 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.868024    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-routing-overlay-volume\") pod \"payload-processing-external-model-59f9b4bfb5-ww55q\" (UID: \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\") " pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"
Nov 02 13:00:59.868174 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.868063    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-plugins-config-volume\") pod \"payload-processing-external-model-59f9b4bfb5-ww55q\" (UID: \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\") " pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"
Nov 02 13:00:59.869576 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.869559    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-provider-credentials\") pod \"payload-processing-external-model-59f9b4bfb5-ww55q\" (UID: \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\") " pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"
Nov 02 13:00:59.929063 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.929038    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-958pn\" (UniqueName: \"kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-kube-api-access-958pn\") pod \"payload-processing-external-model-59f9b4bfb5-ww55q\" (UID: \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\") " pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"
Nov 02 13:00:59.973588 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:00:59.973560    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"
Nov 02 13:01:00.346407 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:00.346376    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"]
Nov 02 13:01:00.347061 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:01:00.347032    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf347e1bb_b9f8_45dc_84fc_66e1fd22ed92.slice/crio-3de4ba2604548ab21edafbbe5e75ccec3e03db0c83643b6855abe4304fff8eb6 WatchSource:0}: Error finding container 3de4ba2604548ab21edafbbe5e75ccec3e03db0c83643b6855abe4304fff8eb6: Status 404 returned error can't find the container with id 3de4ba2604548ab21edafbbe5e75ccec3e03db0c83643b6855abe4304fff8eb6
Nov 02 13:01:00.461502 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:00.461454    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q" event={"ID":"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92","Type":"ContainerStarted","Data":"30ee431321db5bf9d7704f6d1a625f4bc3ab2930bc2a3ff54072360e4cb45c01"}
Nov 02 13:01:00.461636 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:00.461553    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q" event={"ID":"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92","Type":"ContainerStarted","Data":"3de4ba2604548ab21edafbbe5e75ccec3e03db0c83643b6855abe4304fff8eb6"}
Nov 02 13:01:00.461636 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:00.461601    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"
Nov 02 13:01:00.580905 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:00.580853    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q" podStartSLOduration=1.580839765 podStartE2EDuration="1.580839765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:00.580399677 +0000 UTC m=+3889.144108086" watchObservedRunningTime="2026-11-02 13:01:00.580839765 +0000 UTC m=+3889.144548212"
Nov 02 13:01:01.586364 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:01:01.586329    4464 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-897c4765e686: secret "agc-cred-897c4765e686" not found
Nov 02 13:01:01.586364 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:01:01.586359    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q: secret "agc-cred-897c4765e686" not found
Nov 02 13:01:01.586817 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:01:01.586410    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-provider-credentials podName:f347e1bb-b9f8-45dc-84fc-66e1fd22ed92 nodeName:}" failed. No retries permitted until 2026-11-02 13:01:02.086391622 +0000 UTC m=+3890.650100030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-provider-credentials") pod "payload-processing-external-model-59f9b4bfb5-ww55q" (UID: "f347e1bb-b9f8-45dc-84fc-66e1fd22ed92") : secret "agc-cred-897c4765e686" not found
Nov 02 13:01:02.091567 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:01:02.091534    4464 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-897c4765e686: secret "agc-cred-897c4765e686" not found
Nov 02 13:01:02.091567 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:01:02.091561    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q: secret "agc-cred-897c4765e686" not found
Nov 02 13:01:02.091861 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:01:02.091627    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-provider-credentials podName:f347e1bb-b9f8-45dc-84fc-66e1fd22ed92 nodeName:}" failed. No retries permitted until 2026-11-02 13:01:03.091613292 +0000 UTC m=+3891.655321693 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-provider-credentials") pod "payload-processing-external-model-59f9b4bfb5-ww55q" (UID: "f347e1bb-b9f8-45dc-84fc-66e1fd22ed92") : secret "agc-cred-897c4765e686" not found
Nov 02 13:01:02.493188 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:02.493140    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"]
Nov 02 13:01:03.102118 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:01:03.102088    4464 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-897c4765e686: secret "agc-cred-897c4765e686" not found
Nov 02 13:01:03.102118 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:01:03.102114    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q: secret "agc-cred-897c4765e686" not found
Nov 02 13:01:03.102544 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:01:03.102168    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-provider-credentials podName:f347e1bb-b9f8-45dc-84fc-66e1fd22ed92 nodeName:}" failed. No retries permitted until 2026-11-02 13:01:05.102151988 +0000 UTC m=+3893.665860393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-provider-credentials") pod "payload-processing-external-model-59f9b4bfb5-ww55q" (UID: "f347e1bb-b9f8-45dc-84fc-66e1fd22ed92") : secret "agc-cred-897c4765e686" not found
Nov 02 13:01:04.544432 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.544396    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"]
Nov 02 13:01:04.548501 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.548484    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.620418 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.620392    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.620418 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.620428    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.620629 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.620447    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.620677 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.620635    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.620784 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.620760    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e052c7c-9a9d-482d-8757-1b160b965466-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.620930 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.620815    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.721768 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.721736    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.721954 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.721785    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j56xq\" (UniqueName: \"kubernetes.io/projected/2e052c7c-9a9d-482d-8757-1b160b965466-kube-api-access-j56xq\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.721954 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.721834    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.721954 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.721870    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e052c7c-9a9d-482d-8757-1b160b965466-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.722166 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.722061    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.722166 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.722106    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.722296 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.722166    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.722296 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.722202    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.722296 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.722208    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.722476 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.722305    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e052c7c-9a9d-482d-8757-1b160b965466-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.722476 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.722371    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e052c7c-9a9d-482d-8757-1b160b965466-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.722476 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.722438    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.722687 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.722663    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.724364 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.724340    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.724588 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.724570    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e052c7c-9a9d-482d-8757-1b160b965466-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.758506 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.758475    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"]
Nov 02 13:01:04.823439 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.823360    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e052c7c-9a9d-482d-8757-1b160b965466-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.823595 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.823438    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j56xq\" (UniqueName: \"kubernetes.io/projected/2e052c7c-9a9d-482d-8757-1b160b965466-kube-api-access-j56xq\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.823595 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.823495    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e052c7c-9a9d-482d-8757-1b160b965466-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.824171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.824150    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e052c7c-9a9d-482d-8757-1b160b965466-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.885643 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.885612    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e052c7c-9a9d-482d-8757-1b160b965466-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:04.885792 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:04.885773    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j56xq\" (UniqueName: \"kubernetes.io/projected/2e052c7c-9a9d-482d-8757-1b160b965466-kube-api-access-j56xq\") pod \"maas-default-gateway-openshift-default-6dcf757899-xvknm\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:05.127350 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:01:05.127277    4464 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-897c4765e686: secret "agc-cred-897c4765e686" not found
Nov 02 13:01:05.127350 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:01:05.127300    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q: secret "agc-cred-897c4765e686" not found
Nov 02 13:01:05.127350 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:01:05.127352    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-provider-credentials podName:f347e1bb-b9f8-45dc-84fc-66e1fd22ed92 nodeName:}" failed. No retries permitted until 2026-11-02 13:01:09.127336342 +0000 UTC m=+3897.691044748 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-provider-credentials") pod "payload-processing-external-model-59f9b4bfb5-ww55q" (UID: "f347e1bb-b9f8-45dc-84fc-66e1fd22ed92") : secret "agc-cred-897c4765e686" not found
Nov 02 13:01:05.163346 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:05.163316    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:05.643025 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:05.642994    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"]
Nov 02 13:01:05.644572 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:01:05.644527    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e052c7c_9a9d_482d_8757_1b160b965466.slice/crio-842be26033626c7dc85ea4f13cdc57c6be2d7f172d61cfb5bb5dc5e9b3445c7f WatchSource:0}: Error finding container 842be26033626c7dc85ea4f13cdc57c6be2d7f172d61cfb5bb5dc5e9b3445c7f: Status 404 returned error can't find the container with id 842be26033626c7dc85ea4f13cdc57c6be2d7f172d61cfb5bb5dc5e9b3445c7f
Nov 02 13:01:05.646637 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:05.646604    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:01:05.646716 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:05.646677    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:01:05.646761 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:05.646714    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:01:06.501644 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:06.501602    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm" event={"ID":"2e052c7c-9a9d-482d-8757-1b160b965466","Type":"ContainerStarted","Data":"7484deb6afd0d18571bdefcdfca60aad0938a26695848e024085b13c1900fa26"}
Nov 02 13:01:06.501846 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:06.501655    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm" event={"ID":"2e052c7c-9a9d-482d-8757-1b160b965466","Type":"ContainerStarted","Data":"842be26033626c7dc85ea4f13cdc57c6be2d7f172d61cfb5bb5dc5e9b3445c7f"}
Nov 02 13:01:06.686683 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:06.686607    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm" podStartSLOduration=2.686574353 podStartE2EDuration="2.686574353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:06.685336621 +0000 UTC m=+3895.249045054" watchObservedRunningTime="2026-11-02 13:01:06.686574353 +0000 UTC m=+3895.250282765"
Nov 02 13:01:07.166839 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:07.164272    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:07.167777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:07.167747    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-xvknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 13:01:07.167968 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:07.167946    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm" podUID="2e052c7c-9a9d-482d-8757-1b160b965466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 13:01:07.493864 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:07.493799    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q" podUID="f347e1bb-b9f8-45dc-84fc-66e1fd22ed92" containerName="payload-processing" containerID="cri-o://30ee431321db5bf9d7704f6d1a625f4bc3ab2930bc2a3ff54072360e4cb45c01" gracePeriod=25
Nov 02 13:01:07.791683 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:07.791654    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"
Nov 02 13:01:07.882001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:07.881769    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-routing-overlay-volume\") pod \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\" (UID: \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\") "
Nov 02 13:01:07.882001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:07.881840    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-plugins-config-volume\") pod \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\" (UID: \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\") "
Nov 02 13:01:07.882283 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:07.882012    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-provider-credentials\") pod \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\" (UID: \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\") "
Nov 02 13:01:07.882283 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:07.882103    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-kube-api-access-958pn\" (UniqueName: \"kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-kube-api-access-958pn\") pod \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\" (UID: \"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92\") "
Nov 02 13:01:07.884336 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:07.884213    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-routing-overlay-volume" pod "f347e1bb-b9f8-45dc-84fc-66e1fd22ed92" (UID: "f347e1bb-b9f8-45dc-84fc-66e1fd22ed92"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:01:07.884504 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:07.884461    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-plugins-config-volume" pod "f347e1bb-b9f8-45dc-84fc-66e1fd22ed92" (UID: "f347e1bb-b9f8-45dc-84fc-66e1fd22ed92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:01:07.886350 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:07.886325    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-provider-credentials" pod "f347e1bb-b9f8-45dc-84fc-66e1fd22ed92" (UID: "f347e1bb-b9f8-45dc-84fc-66e1fd22ed92"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:01:07.886459 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:07.886384    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-kube-api-access-958pn" pod "f347e1bb-b9f8-45dc-84fc-66e1fd22ed92" (UID: "f347e1bb-b9f8-45dc-84fc-66e1fd22ed92"). InnerVolumeSpecName "kube-api-access-958pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:01:07.984752 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:07.984710    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:01:07.984752 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:07.984744    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-958pn\" (UniqueName: \"kubernetes.io/projected/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-kube-api-access-958pn\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:01:07.984752 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:07.984756    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:01:07.985149 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:07.984767    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:01:08.179053 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:08.171376    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-xvknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 13:01:08.179053 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:08.171439    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm" podUID="2e052c7c-9a9d-482d-8757-1b160b965466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 13:01:08.513178 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:08.513145    4464 generic.go:356] "Generic (PLEG): container finished" podID="f347e1bb-b9f8-45dc-84fc-66e1fd22ed92" containerID="30ee431321db5bf9d7704f6d1a625f4bc3ab2930bc2a3ff54072360e4cb45c01" exitCode=0
Nov 02 13:01:08.513406 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:08.513198    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"
Nov 02 13:01:08.513406 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:08.513236    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q" event={"ID":"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92","Type":"ContainerDied","Data":"30ee431321db5bf9d7704f6d1a625f4bc3ab2930bc2a3ff54072360e4cb45c01"}
Nov 02 13:01:08.513406 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:08.513298    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q" event={"ID":"f347e1bb-b9f8-45dc-84fc-66e1fd22ed92","Type":"ContainerDied","Data":"3de4ba2604548ab21edafbbe5e75ccec3e03db0c83643b6855abe4304fff8eb6"}
Nov 02 13:01:08.513406 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:08.513327    4464 scope.go:122] "RemoveContainer" containerID="30ee431321db5bf9d7704f6d1a625f4bc3ab2930bc2a3ff54072360e4cb45c01"
Nov 02 13:01:08.525872 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:08.525549    4464 scope.go:122] "RemoveContainer" containerID="30ee431321db5bf9d7704f6d1a625f4bc3ab2930bc2a3ff54072360e4cb45c01"
Nov 02 13:01:08.526401 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:01:08.526088    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30ee431321db5bf9d7704f6d1a625f4bc3ab2930bc2a3ff54072360e4cb45c01\": container with ID starting with 30ee431321db5bf9d7704f6d1a625f4bc3ab2930bc2a3ff54072360e4cb45c01 not found: ID does not exist" containerID="30ee431321db5bf9d7704f6d1a625f4bc3ab2930bc2a3ff54072360e4cb45c01"
Nov 02 13:01:08.526401 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:08.526117    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30ee431321db5bf9d7704f6d1a625f4bc3ab2930bc2a3ff54072360e4cb45c01"} err="failed to get container status \"30ee431321db5bf9d7704f6d1a625f4bc3ab2930bc2a3ff54072360e4cb45c01\": rpc error: code = NotFound desc = could not find container \"30ee431321db5bf9d7704f6d1a625f4bc3ab2930bc2a3ff54072360e4cb45c01\": container with ID starting with 30ee431321db5bf9d7704f6d1a625f4bc3ab2930bc2a3ff54072360e4cb45c01 not found: ID does not exist"
Nov 02 13:01:08.682002 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:08.680370    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:01:08.705167 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:08.705084    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"]
Nov 02 13:01:08.724226 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:08.722283    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-59f9b4bfb5-ww55q"]
Nov 02 13:01:09.173999 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:09.164421    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-xvknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 13:01:09.173999 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:09.164491    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm" podUID="2e052c7c-9a9d-482d-8757-1b160b965466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 13:01:09.589890 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:09.589859    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f347e1bb-b9f8-45dc-84fc-66e1fd22ed92" path="/var/lib/kubelet/pods/f347e1bb-b9f8-45dc-84fc-66e1fd22ed92/volumes"
Nov 02 13:01:10.180000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.170378    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-xvknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 13:01:10.180000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.170443    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm" podUID="2e052c7c-9a9d-482d-8757-1b160b965466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 13:01:10.242569 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.242535    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"]
Nov 02 13:01:10.243126 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.243111    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f347e1bb-b9f8-45dc-84fc-66e1fd22ed92" containerName="payload-processing"
Nov 02 13:01:10.243180 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.243136    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f347e1bb-b9f8-45dc-84fc-66e1fd22ed92" containerName="payload-processing"
Nov 02 13:01:10.243252 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.243242    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f347e1bb-b9f8-45dc-84fc-66e1fd22ed92" containerName="payload-processing"
Nov 02 13:01:10.247844 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.247826    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:01:10.304048 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.304009    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"]
Nov 02 13:01:10.327638 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.327603    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11058bd3-db12-4168-9707-eb0431e3754d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85\" (UID: \"11058bd3-db12-4168-9707-eb0431e3754d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:01:10.327830 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.327685    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdrwn\" (UniqueName: \"kubernetes.io/projected/11058bd3-db12-4168-9707-eb0431e3754d-kube-api-access-xdrwn\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85\" (UID: \"11058bd3-db12-4168-9707-eb0431e3754d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:01:10.327830 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.327713    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11058bd3-db12-4168-9707-eb0431e3754d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85\" (UID: \"11058bd3-db12-4168-9707-eb0431e3754d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:01:10.327830 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.327739    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11058bd3-db12-4168-9707-eb0431e3754d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85\" (UID: \"11058bd3-db12-4168-9707-eb0431e3754d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:01:10.335221 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.335193    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"kube-root-ca.crt\""
Nov 02 13:01:10.351900 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.351876    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"agc-cred-1b6e08df785d\""
Nov 02 13:01:10.352080 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.351945    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"routing-overlay\""
Nov 02 13:01:10.352080 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.351971    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"openshift-service-ca.crt\""
Nov 02 13:01:10.352080 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.352052    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"payload-processing-external-model-e2e--75a83f02-dockercfg-24d26\""
Nov 02 13:01:10.352248 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.352093    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6ac03f\""
Nov 02 13:01:10.429253 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.429216    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdrwn\" (UniqueName: \"kubernetes.io/projected/11058bd3-db12-4168-9707-eb0431e3754d-kube-api-access-xdrwn\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85\" (UID: \"11058bd3-db12-4168-9707-eb0431e3754d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:01:10.429461 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.429261    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11058bd3-db12-4168-9707-eb0431e3754d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85\" (UID: \"11058bd3-db12-4168-9707-eb0431e3754d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:01:10.429461 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.429385    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11058bd3-db12-4168-9707-eb0431e3754d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85\" (UID: \"11058bd3-db12-4168-9707-eb0431e3754d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:01:10.429595 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.429473    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11058bd3-db12-4168-9707-eb0431e3754d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85\" (UID: \"11058bd3-db12-4168-9707-eb0431e3754d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:01:10.429968 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.429943    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11058bd3-db12-4168-9707-eb0431e3754d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85\" (UID: \"11058bd3-db12-4168-9707-eb0431e3754d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:01:10.430306 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.430255    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11058bd3-db12-4168-9707-eb0431e3754d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85\" (UID: \"11058bd3-db12-4168-9707-eb0431e3754d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:01:10.435344 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.435324    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11058bd3-db12-4168-9707-eb0431e3754d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85\" (UID: \"11058bd3-db12-4168-9707-eb0431e3754d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:01:10.501458 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.501424    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdrwn\" (UniqueName: \"kubernetes.io/projected/11058bd3-db12-4168-9707-eb0431e3754d-kube-api-access-xdrwn\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85\" (UID: \"11058bd3-db12-4168-9707-eb0431e3754d\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:01:10.576203 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.576171    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:01:10.981255 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:10.981149    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"]
Nov 02 13:01:10.986333 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:01:10.986303    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11058bd3_db12_4168_9707_eb0431e3754d.slice/crio-e2630c019ecf07b34579986db044a1c078f51a96ee9611ef4769e86e844d6d7b WatchSource:0}: Error finding container e2630c019ecf07b34579986db044a1c078f51a96ee9611ef4769e86e844d6d7b: Status 404 returned error can't find the container with id e2630c019ecf07b34579986db044a1c078f51a96ee9611ef4769e86e844d6d7b
Nov 02 13:01:11.179372 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:11.179335    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-xvknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 13:01:11.179627 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:11.179599    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm" podUID="2e052c7c-9a9d-482d-8757-1b160b965466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 13:01:11.531905 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:11.531874    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85" event={"ID":"11058bd3-db12-4168-9707-eb0431e3754d","Type":"ContainerStarted","Data":"027839d9e4d5412f3918697be611233ab8e597520e7205d4dc5855886cd067bb"}
Nov 02 13:01:11.532347 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:11.531923    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85" event={"ID":"11058bd3-db12-4168-9707-eb0431e3754d","Type":"ContainerStarted","Data":"e2630c019ecf07b34579986db044a1c078f51a96ee9611ef4769e86e844d6d7b"}
Nov 02 13:01:11.532347 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:11.531971    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:01:11.654884 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:11.654837    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85" podStartSLOduration=1.6548248349999999 podStartE2EDuration="1.654824835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:11.652954039 +0000 UTC m=+3900.216662461" watchObservedRunningTime="2026-11-02 13:01:11.654824835 +0000 UTC m=+3900.218533244"
Nov 02 13:01:12.093094 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:12.093065    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:01:12.111820 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:12.111790    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:01:12.133533 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:12.133512    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:01:12.152813 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:12.152788    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:01:12.163814 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:12.163791    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-xvknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 13:01:12.163930 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:12.163837    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm" podUID="2e052c7c-9a9d-482d-8757-1b160b965466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 13:01:13.164665 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:13.164632    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-xvknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Nov 02 13:01:13.165088 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:13.164699    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm" podUID="2e052c7c-9a9d-482d-8757-1b160b965466" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Nov 02 13:01:14.933034 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:14.933001    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:14.933522 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:14.933488    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:14.935644 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:14.935621    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:01:22.541712 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:22.541680    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:01:24.779213 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:24.779138    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:01:28.291940 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:01:28.291901    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:01:33.975353 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:33.975316    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"]
Nov 02 13:01:33.979220 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:33.979199    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:01:34.074284 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.074253    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4\" (UID: \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:01:34.074451 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.074295    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gtdj\" (UniqueName: \"kubernetes.io/projected/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-kube-api-access-7gtdj\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4\" (UID: \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:01:34.074451 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.074328    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4\" (UID: \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:01:34.074528 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.074483    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4\" (UID: \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:01:34.085456 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.085423    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"openshift-service-ca.crt\""
Nov 02 13:01:34.085456 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.085423    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"routing-overlay\""
Nov 02 13:01:34.085606 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.085427    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3a78b4\""
Nov 02 13:01:34.085606 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.085475    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"kube-root-ca.crt\""
Nov 02 13:01:34.087740 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.087718    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"]
Nov 02 13:01:34.104624 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.104605    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-cb4360bb5351\""
Nov 02 13:01:34.104897 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.104882    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"payload-processing-external-model-e2e--a8ab13a3-dockercfg-b8qnc\""
Nov 02 13:01:34.175915 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.175884    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4\" (UID: \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:01:34.176088 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.175926    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gtdj\" (UniqueName: \"kubernetes.io/projected/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-kube-api-access-7gtdj\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4\" (UID: \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:01:34.176088 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.176004    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4\" (UID: \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:01:34.176214 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.176100    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4\" (UID: \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:01:34.176731 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.176713    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4\" (UID: \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:01:34.176779 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.176725    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4\" (UID: \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:01:34.178502 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.178473    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4\" (UID: \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:01:34.237820 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.237761    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gtdj\" (UniqueName: \"kubernetes.io/projected/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-kube-api-access-7gtdj\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4\" (UID: \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:01:34.291778 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.291753    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:01:34.663055 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:34.663021    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"]
Nov 02 13:01:34.664285 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:01:34.664257    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0fbb5a_0637_4743_ac99_e4d2fb34feb7.slice/crio-9c7a31e05ccc51545cccc987c330585c9471621455272ea0b366d1923d767c5e WatchSource:0}: Error finding container 9c7a31e05ccc51545cccc987c330585c9471621455272ea0b366d1923d767c5e: Status 404 returned error can't find the container with id 9c7a31e05ccc51545cccc987c330585c9471621455272ea0b366d1923d767c5e
Nov 02 13:01:35.664183 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:35.664141    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4" event={"ID":"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7","Type":"ContainerStarted","Data":"bb2f45b3aa3e2bc465fed0c2d8e649c29fc26b798b38a84aa0650701e727d3a0"}
Nov 02 13:01:35.664669 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:35.664195    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4" event={"ID":"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7","Type":"ContainerStarted","Data":"9c7a31e05ccc51545cccc987c330585c9471621455272ea0b366d1923d767c5e"}
Nov 02 13:01:35.664669 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:35.664234    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:01:35.784728 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:35.784680    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4" podStartSLOduration=2.784664736 podStartE2EDuration="2.784664736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:35.783248876 +0000 UTC m=+3924.346957286" watchObservedRunningTime="2026-11-02 13:01:35.784664736 +0000 UTC m=+3924.348373145"
Nov 02 13:01:39.281048 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:39.281013    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:01:46.672950 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:46.672922    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:01:49.672170 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:49.670451    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:01:54.654303 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:54.654270    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"]
Nov 02 13:01:54.658234 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:54.658211    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"
Nov 02 13:01:54.711901 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:54.711873    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:01:54.711901 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:54.711903    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:01:54.712100 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:54.711990    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:01:54.766159 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:54.766124    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 13:01:54.766607 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:54.766583    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"]
Nov 02 13:01:54.818654 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:54.818617    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ba84a2-91ce-4da5-bee6-db5eb46225f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64aa50e7-fqp22\" (UID: \"91ba84a2-91ce-4da5-bee6-db5eb46225f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"
Nov 02 13:01:54.818804 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:54.818658    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s4hk\" (UniqueName: \"kubernetes.io/projected/91ba84a2-91ce-4da5-bee6-db5eb46225f2-kube-api-access-8s4hk\") pod \"maas-api-revoke-sub-64aa50e7-fqp22\" (UID: \"91ba84a2-91ce-4da5-bee6-db5eb46225f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"
Nov 02 13:01:54.818804 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:54.818706    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ba84a2-91ce-4da5-bee6-db5eb46225f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64aa50e7-fqp22\" (UID: \"91ba84a2-91ce-4da5-bee6-db5eb46225f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"
Nov 02 13:01:54.919731 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:54.919661    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ba84a2-91ce-4da5-bee6-db5eb46225f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64aa50e7-fqp22\" (UID: \"91ba84a2-91ce-4da5-bee6-db5eb46225f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"
Nov 02 13:01:54.919731 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:54.919695    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s4hk\" (UniqueName: \"kubernetes.io/projected/91ba84a2-91ce-4da5-bee6-db5eb46225f2-kube-api-access-8s4hk\") pod \"maas-api-revoke-sub-64aa50e7-fqp22\" (UID: \"91ba84a2-91ce-4da5-bee6-db5eb46225f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"
Nov 02 13:01:54.919731 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:54.919720    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ba84a2-91ce-4da5-bee6-db5eb46225f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64aa50e7-fqp22\" (UID: \"91ba84a2-91ce-4da5-bee6-db5eb46225f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"
Nov 02 13:01:54.920459 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:54.920432    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ba84a2-91ce-4da5-bee6-db5eb46225f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64aa50e7-fqp22\" (UID: \"91ba84a2-91ce-4da5-bee6-db5eb46225f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"
Nov 02 13:01:54.922090 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:54.922069    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ba84a2-91ce-4da5-bee6-db5eb46225f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64aa50e7-fqp22\" (UID: \"91ba84a2-91ce-4da5-bee6-db5eb46225f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"
Nov 02 13:01:54.981713 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:54.981689    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s4hk\" (UniqueName: \"kubernetes.io/projected/91ba84a2-91ce-4da5-bee6-db5eb46225f2-kube-api-access-8s4hk\") pod \"maas-api-revoke-sub-64aa50e7-fqp22\" (UID: \"91ba84a2-91ce-4da5-bee6-db5eb46225f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"
Nov 02 13:01:55.271362 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:55.271331    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"
Nov 02 13:01:55.645433 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:55.645404    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"]
Nov 02 13:01:55.646419 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:01:55.646392    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ba84a2_91ce_4da5_bee6_db5eb46225f2.slice/crio-9ec0046874090bad4e9aa62723bfd7b8f697c413c968ac9c91228467e564bfb1 WatchSource:0}: Error finding container 9ec0046874090bad4e9aa62723bfd7b8f697c413c968ac9c91228467e564bfb1: Status 404 returned error can't find the container with id 9ec0046874090bad4e9aa62723bfd7b8f697c413c968ac9c91228467e564bfb1
Nov 02 13:01:55.782491 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:55.782458    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22" event={"ID":"91ba84a2-91ce-4da5-bee6-db5eb46225f2","Type":"ContainerStarted","Data":"9ec0046874090bad4e9aa62723bfd7b8f697c413c968ac9c91228467e564bfb1"}
Nov 02 13:01:56.796463 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:56.796429    4464 generic.go:356] "Generic (PLEG): container finished" podID="91ba84a2-91ce-4da5-bee6-db5eb46225f2" containerID="b86031be795ba03a15e968d027404f21bb4e072b9f54601c194c49ed010081bb" exitCode=0
Nov 02 13:01:56.796856 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:56.796504    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22" event={"ID":"91ba84a2-91ce-4da5-bee6-db5eb46225f2","Type":"ContainerDied","Data":"b86031be795ba03a15e968d027404f21bb4e072b9f54601c194c49ed010081bb"}
Nov 02 13:01:57.938414 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:57.938391    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"
Nov 02 13:01:58.047987 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:58.047950    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ba84a2-91ce-4da5-bee6-db5eb46225f2-kube-api-access-8s4hk\" (UniqueName: \"kubernetes.io/projected/91ba84a2-91ce-4da5-bee6-db5eb46225f2-kube-api-access-8s4hk\") pod \"91ba84a2-91ce-4da5-bee6-db5eb46225f2\" (UID: \"91ba84a2-91ce-4da5-bee6-db5eb46225f2\") "
Nov 02 13:01:58.048117 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:58.048023    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ba84a2-91ce-4da5-bee6-db5eb46225f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ba84a2-91ce-4da5-bee6-db5eb46225f2-maas-api-openshift-ca\") pod \"91ba84a2-91ce-4da5-bee6-db5eb46225f2\" (UID: \"91ba84a2-91ce-4da5-bee6-db5eb46225f2\") "
Nov 02 13:01:58.048117 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:58.048096    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91ba84a2-91ce-4da5-bee6-db5eb46225f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ba84a2-91ce-4da5-bee6-db5eb46225f2-maas-api-xks-ca\") pod \"91ba84a2-91ce-4da5-bee6-db5eb46225f2\" (UID: \"91ba84a2-91ce-4da5-bee6-db5eb46225f2\") "
Nov 02 13:01:58.048335 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:58.048313    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ba84a2-91ce-4da5-bee6-db5eb46225f2-maas-api-openshift-ca" pod "91ba84a2-91ce-4da5-bee6-db5eb46225f2" (UID: "91ba84a2-91ce-4da5-bee6-db5eb46225f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:01:58.049962 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:58.049936    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ba84a2-91ce-4da5-bee6-db5eb46225f2-maas-api-xks-ca" pod "91ba84a2-91ce-4da5-bee6-db5eb46225f2" (UID: "91ba84a2-91ce-4da5-bee6-db5eb46225f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:01:58.050080 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:58.049987    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ba84a2-91ce-4da5-bee6-db5eb46225f2-kube-api-access-8s4hk" pod "91ba84a2-91ce-4da5-bee6-db5eb46225f2" (UID: "91ba84a2-91ce-4da5-bee6-db5eb46225f2"). InnerVolumeSpecName "kube-api-access-8s4hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:01:58.149500 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:58.149472    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s4hk\" (UniqueName: \"kubernetes.io/projected/91ba84a2-91ce-4da5-bee6-db5eb46225f2-kube-api-access-8s4hk\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:01:58.149500 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:58.149498    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ba84a2-91ce-4da5-bee6-db5eb46225f2-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:01:58.149658 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:58.149508    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ba84a2-91ce-4da5-bee6-db5eb46225f2-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:01:58.291922 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:01:58.291891    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:01:58.808922 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:58.808894    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"
Nov 02 13:01:58.809111 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:58.808891    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22" event={"ID":"91ba84a2-91ce-4da5-bee6-db5eb46225f2","Type":"ContainerDied","Data":"9ec0046874090bad4e9aa62723bfd7b8f697c413c968ac9c91228467e564bfb1"}
Nov 02 13:01:58.809111 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:01:58.809039    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ec0046874090bad4e9aa62723bfd7b8f697c413c968ac9c91228467e564bfb1"
Nov 02 13:02:07.027726 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.027692    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"]
Nov 02 13:02:07.843943 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.843903    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"]
Nov 02 13:02:07.844402 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.844389    4464 cpu_manager.go:414] "containerMap: removing container" podUID="91ba84a2-91ce-4da5-bee6-db5eb46225f2" containerName="revoke-keys"
Nov 02 13:02:07.844453 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.844404    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ba84a2-91ce-4da5-bee6-db5eb46225f2" containerName="revoke-keys"
Nov 02 13:02:07.844509 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.844500    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ba84a2-91ce-4da5-bee6-db5eb46225f2" containerName="revoke-keys"
Nov 02 13:02:07.847880 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.847863    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"
Nov 02 13:02:07.870381 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.870347    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0f2a23-6h8xm\" (UID: \"4e37cbdd-16a5-4d30-865c-fee2a8d277a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"
Nov 02 13:02:07.870551 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.870411    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0f2a23-6h8xm\" (UID: \"4e37cbdd-16a5-4d30-865c-fee2a8d277a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"
Nov 02 13:02:07.870551 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.870502    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh4wt\" (UniqueName: \"kubernetes.io/projected/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-kube-api-access-gh4wt\") pod \"maas-api-revoke-sub-cd0f2a23-6h8xm\" (UID: \"4e37cbdd-16a5-4d30-865c-fee2a8d277a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"
Nov 02 13:02:07.880738 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.880711    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:02:07.903248 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.903224    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:02:07.903420 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.903398    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:02:07.903591 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.903543    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 13:02:07.903699 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.903683    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:02:07.932719 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.932693    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"]
Nov 02 13:02:07.979295 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.979255    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0f2a23-6h8xm\" (UID: \"4e37cbdd-16a5-4d30-865c-fee2a8d277a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"
Nov 02 13:02:07.979450 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.979325    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0f2a23-6h8xm\" (UID: \"4e37cbdd-16a5-4d30-865c-fee2a8d277a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"
Nov 02 13:02:07.979450 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.979359    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh4wt\" (UniqueName: \"kubernetes.io/projected/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-kube-api-access-gh4wt\") pod \"maas-api-revoke-sub-cd0f2a23-6h8xm\" (UID: \"4e37cbdd-16a5-4d30-865c-fee2a8d277a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"
Nov 02 13:02:07.982999 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.982952    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0f2a23-6h8xm\" (UID: \"4e37cbdd-16a5-4d30-865c-fee2a8d277a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"
Nov 02 13:02:07.983622 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:07.983602    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0f2a23-6h8xm\" (UID: \"4e37cbdd-16a5-4d30-865c-fee2a8d277a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"
Nov 02 13:02:08.041384 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:08.041352    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh4wt\" (UniqueName: \"kubernetes.io/projected/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-kube-api-access-gh4wt\") pod \"maas-api-revoke-sub-cd0f2a23-6h8xm\" (UID: \"4e37cbdd-16a5-4d30-865c-fee2a8d277a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"
Nov 02 13:02:08.160597 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:08.160532    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"
Nov 02 13:02:08.532592 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:08.532563    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"]
Nov 02 13:02:08.533828 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:02:08.533803    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e37cbdd_16a5_4d30_865c_fee2a8d277a0.slice/crio-e738576a30bd190da2b93fc484602348ad2ba377e11df8152e11c22e35a17371 WatchSource:0}: Error finding container e738576a30bd190da2b93fc484602348ad2ba377e11df8152e11c22e35a17371: Status 404 returned error can't find the container with id e738576a30bd190da2b93fc484602348ad2ba377e11df8152e11c22e35a17371
Nov 02 13:02:08.869476 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:08.869448    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm" event={"ID":"4e37cbdd-16a5-4d30-865c-fee2a8d277a0","Type":"ContainerStarted","Data":"e738576a30bd190da2b93fc484602348ad2ba377e11df8152e11c22e35a17371"}
Nov 02 13:02:09.876078 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:09.876044    4464 generic.go:356] "Generic (PLEG): container finished" podID="4e37cbdd-16a5-4d30-865c-fee2a8d277a0" containerID="1e27b8d6f7dd6387329d2b999881eed3d231c230e5da41181d321be60910736f" exitCode=0
Nov 02 13:02:09.876474 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:09.876101    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm" event={"ID":"4e37cbdd-16a5-4d30-865c-fee2a8d277a0","Type":"ContainerDied","Data":"1e27b8d6f7dd6387329d2b999881eed3d231c230e5da41181d321be60910736f"}
Nov 02 13:02:11.017887 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:11.017864    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"
Nov 02 13:02:11.111489 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:11.111453    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-maas-api-xks-ca\") pod \"4e37cbdd-16a5-4d30-865c-fee2a8d277a0\" (UID: \"4e37cbdd-16a5-4d30-865c-fee2a8d277a0\") "
Nov 02 13:02:11.111678 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:11.111512    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-maas-api-openshift-ca\") pod \"4e37cbdd-16a5-4d30-865c-fee2a8d277a0\" (UID: \"4e37cbdd-16a5-4d30-865c-fee2a8d277a0\") "
Nov 02 13:02:11.111678 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:11.111625    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-kube-api-access-gh4wt\" (UniqueName: \"kubernetes.io/projected/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-kube-api-access-gh4wt\") pod \"4e37cbdd-16a5-4d30-865c-fee2a8d277a0\" (UID: \"4e37cbdd-16a5-4d30-865c-fee2a8d277a0\") "
Nov 02 13:02:11.111848 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:11.111830    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-maas-api-openshift-ca" pod "4e37cbdd-16a5-4d30-865c-fee2a8d277a0" (UID: "4e37cbdd-16a5-4d30-865c-fee2a8d277a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:02:11.112030 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:11.112016    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:02:11.113558 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:11.113536    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-maas-api-xks-ca" pod "4e37cbdd-16a5-4d30-865c-fee2a8d277a0" (UID: "4e37cbdd-16a5-4d30-865c-fee2a8d277a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:02:11.113685 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:11.113664    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-kube-api-access-gh4wt" pod "4e37cbdd-16a5-4d30-865c-fee2a8d277a0" (UID: "4e37cbdd-16a5-4d30-865c-fee2a8d277a0"). InnerVolumeSpecName "kube-api-access-gh4wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:02:11.212826 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:11.212748    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:02:11.212826 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:11.212774    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh4wt\" (UniqueName: \"kubernetes.io/projected/4e37cbdd-16a5-4d30-865c-fee2a8d277a0-kube-api-access-gh4wt\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:02:11.888115 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:11.888037    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"
Nov 02 13:02:11.888115 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:11.888061    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm" event={"ID":"4e37cbdd-16a5-4d30-865c-fee2a8d277a0","Type":"ContainerDied","Data":"e738576a30bd190da2b93fc484602348ad2ba377e11df8152e11c22e35a17371"}
Nov 02 13:02:11.888115 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:11.888107    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e738576a30bd190da2b93fc484602348ad2ba377e11df8152e11c22e35a17371"
Nov 02 13:02:12.028043 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.028001    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4" podUID="6a0fbb5a-0637-4743-ac99-e4d2fb34feb7" containerName="payload-processing" containerID="cri-o://bb2f45b3aa3e2bc465fed0c2d8e649c29fc26b798b38a84aa0650701e727d3a0" gracePeriod=25
Nov 02 13:02:12.281543 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.281504    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:02:12.324870 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.324836    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-plugins-config-volume\") pod \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\" (UID: \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\") "
Nov 02 13:02:12.324870 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.324871    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-provider-credentials\") pod \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\" (UID: \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\") "
Nov 02 13:02:12.325177 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.324949    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-kube-api-access-7gtdj\" (UniqueName: \"kubernetes.io/projected/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-kube-api-access-7gtdj\") pod \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\" (UID: \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\") "
Nov 02 13:02:12.325177 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.324993    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-routing-overlay-volume\") pod \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\" (UID: \"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7\") "
Nov 02 13:02:12.325372 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.325343    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-plugins-config-volume" pod "6a0fbb5a-0637-4743-ac99-e4d2fb34feb7" (UID: "6a0fbb5a-0637-4743-ac99-e4d2fb34feb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:02:12.325494 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.325397    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-routing-overlay-volume" pod "6a0fbb5a-0637-4743-ac99-e4d2fb34feb7" (UID: "6a0fbb5a-0637-4743-ac99-e4d2fb34feb7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:02:12.327153 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.327128    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-kube-api-access-7gtdj" pod "6a0fbb5a-0637-4743-ac99-e4d2fb34feb7" (UID: "6a0fbb5a-0637-4743-ac99-e4d2fb34feb7"). InnerVolumeSpecName "kube-api-access-7gtdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:02:12.327244 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.327149    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-provider-credentials" pod "6a0fbb5a-0637-4743-ac99-e4d2fb34feb7" (UID: "6a0fbb5a-0637-4743-ac99-e4d2fb34feb7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:02:12.426208 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.426129    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:02:12.426208 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.426156    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:02:12.426208 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.426165    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gtdj\" (UniqueName: \"kubernetes.io/projected/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-kube-api-access-7gtdj\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:02:12.426208 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.426174    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:02:12.895129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.895094    4464 generic.go:356] "Generic (PLEG): container finished" podID="6a0fbb5a-0637-4743-ac99-e4d2fb34feb7" containerID="bb2f45b3aa3e2bc465fed0c2d8e649c29fc26b798b38a84aa0650701e727d3a0" exitCode=0
Nov 02 13:02:12.895319 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.895156    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"
Nov 02 13:02:12.895319 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.895165    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4" event={"ID":"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7","Type":"ContainerDied","Data":"bb2f45b3aa3e2bc465fed0c2d8e649c29fc26b798b38a84aa0650701e727d3a0"}
Nov 02 13:02:12.895319 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.895234    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4" event={"ID":"6a0fbb5a-0637-4743-ac99-e4d2fb34feb7","Type":"ContainerDied","Data":"9c7a31e05ccc51545cccc987c330585c9471621455272ea0b366d1923d767c5e"}
Nov 02 13:02:12.895319 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.895256    4464 scope.go:122] "RemoveContainer" containerID="bb2f45b3aa3e2bc465fed0c2d8e649c29fc26b798b38a84aa0650701e727d3a0"
Nov 02 13:02:12.904942 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.904924    4464 scope.go:122] "RemoveContainer" containerID="bb2f45b3aa3e2bc465fed0c2d8e649c29fc26b798b38a84aa0650701e727d3a0"
Nov 02 13:02:12.905295 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:02:12.905275    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2f45b3aa3e2bc465fed0c2d8e649c29fc26b798b38a84aa0650701e727d3a0\": container with ID starting with bb2f45b3aa3e2bc465fed0c2d8e649c29fc26b798b38a84aa0650701e727d3a0 not found: ID does not exist" containerID="bb2f45b3aa3e2bc465fed0c2d8e649c29fc26b798b38a84aa0650701e727d3a0"
Nov 02 13:02:12.905344 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:12.905305    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2f45b3aa3e2bc465fed0c2d8e649c29fc26b798b38a84aa0650701e727d3a0"} err="failed to get container status \"bb2f45b3aa3e2bc465fed0c2d8e649c29fc26b798b38a84aa0650701e727d3a0\": rpc error: code = NotFound desc = could not find container \"bb2f45b3aa3e2bc465fed0c2d8e649c29fc26b798b38a84aa0650701e727d3a0\": container with ID starting with bb2f45b3aa3e2bc465fed0c2d8e649c29fc26b798b38a84aa0650701e727d3a0 not found: ID does not exist"
Nov 02 13:02:13.063315 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:13.063281    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"]
Nov 02 13:02:13.113993 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:13.113952    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-564c75vv4"]
Nov 02 13:02:13.580863 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:13.580832    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0fbb5a-0637-4743-ac99-e4d2fb34feb7" path="/var/lib/kubelet/pods/6a0fbb5a-0637-4743-ac99-e4d2fb34feb7/volumes"
Nov 02 13:02:18.577483 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:18.577449    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:02:20.171605 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:20.171567    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"]
Nov 02 13:02:25.172732 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.172670    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85" podUID="11058bd3-db12-4168-9707-eb0431e3754d" containerName="payload-processing" containerID="cri-o://027839d9e4d5412f3918697be611233ab8e597520e7205d4dc5855886cd067bb" gracePeriod=25
Nov 02 13:02:25.420516 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.420486    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:02:25.457475 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.457409    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11058bd3-db12-4168-9707-eb0431e3754d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11058bd3-db12-4168-9707-eb0431e3754d-plugins-config-volume\") pod \"11058bd3-db12-4168-9707-eb0431e3754d\" (UID: \"11058bd3-db12-4168-9707-eb0431e3754d\") "
Nov 02 13:02:25.457597 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.457564    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11058bd3-db12-4168-9707-eb0431e3754d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/11058bd3-db12-4168-9707-eb0431e3754d-provider-credentials\") pod \"11058bd3-db12-4168-9707-eb0431e3754d\" (UID: \"11058bd3-db12-4168-9707-eb0431e3754d\") "
Nov 02 13:02:25.457668 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.457621    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11058bd3-db12-4168-9707-eb0431e3754d-kube-api-access-xdrwn\" (UniqueName: \"kubernetes.io/projected/11058bd3-db12-4168-9707-eb0431e3754d-kube-api-access-xdrwn\") pod \"11058bd3-db12-4168-9707-eb0431e3754d\" (UID: \"11058bd3-db12-4168-9707-eb0431e3754d\") "
Nov 02 13:02:25.457668 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.457657    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11058bd3-db12-4168-9707-eb0431e3754d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11058bd3-db12-4168-9707-eb0431e3754d-routing-overlay-volume\") pod \"11058bd3-db12-4168-9707-eb0431e3754d\" (UID: \"11058bd3-db12-4168-9707-eb0431e3754d\") "
Nov 02 13:02:25.457782 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.457732    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11058bd3-db12-4168-9707-eb0431e3754d-plugins-config-volume" pod "11058bd3-db12-4168-9707-eb0431e3754d" (UID: "11058bd3-db12-4168-9707-eb0431e3754d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:02:25.457997 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.457958    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11058bd3-db12-4168-9707-eb0431e3754d-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:02:25.458080 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.458062    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11058bd3-db12-4168-9707-eb0431e3754d-routing-overlay-volume" pod "11058bd3-db12-4168-9707-eb0431e3754d" (UID: "11058bd3-db12-4168-9707-eb0431e3754d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:02:25.459705 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.459678    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11058bd3-db12-4168-9707-eb0431e3754d-kube-api-access-xdrwn" pod "11058bd3-db12-4168-9707-eb0431e3754d" (UID: "11058bd3-db12-4168-9707-eb0431e3754d"). InnerVolumeSpecName "kube-api-access-xdrwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:02:25.459821 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.459714    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11058bd3-db12-4168-9707-eb0431e3754d-provider-credentials" pod "11058bd3-db12-4168-9707-eb0431e3754d" (UID: "11058bd3-db12-4168-9707-eb0431e3754d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:02:25.559457 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.559421    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11058bd3-db12-4168-9707-eb0431e3754d-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:02:25.559457 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.559456    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdrwn\" (UniqueName: \"kubernetes.io/projected/11058bd3-db12-4168-9707-eb0431e3754d-kube-api-access-xdrwn\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:02:25.559712 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.559474    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11058bd3-db12-4168-9707-eb0431e3754d-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:02:25.968032 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.967996    4464 generic.go:356] "Generic (PLEG): container finished" podID="11058bd3-db12-4168-9707-eb0431e3754d" containerID="027839d9e4d5412f3918697be611233ab8e597520e7205d4dc5855886cd067bb" exitCode=0
Nov 02 13:02:25.968210 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.968039    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"
Nov 02 13:02:25.968210 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.968076    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85" event={"ID":"11058bd3-db12-4168-9707-eb0431e3754d","Type":"ContainerDied","Data":"027839d9e4d5412f3918697be611233ab8e597520e7205d4dc5855886cd067bb"}
Nov 02 13:02:25.968210 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.968128    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85" event={"ID":"11058bd3-db12-4168-9707-eb0431e3754d","Type":"ContainerDied","Data":"e2630c019ecf07b34579986db044a1c078f51a96ee9611ef4769e86e844d6d7b"}
Nov 02 13:02:25.968210 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.968149    4464 scope.go:122] "RemoveContainer" containerID="027839d9e4d5412f3918697be611233ab8e597520e7205d4dc5855886cd067bb"
Nov 02 13:02:25.976772 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.976753    4464 scope.go:122] "RemoveContainer" containerID="027839d9e4d5412f3918697be611233ab8e597520e7205d4dc5855886cd067bb"
Nov 02 13:02:25.977023 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:02:25.977007    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027839d9e4d5412f3918697be611233ab8e597520e7205d4dc5855886cd067bb\": container with ID starting with 027839d9e4d5412f3918697be611233ab8e597520e7205d4dc5855886cd067bb not found: ID does not exist" containerID="027839d9e4d5412f3918697be611233ab8e597520e7205d4dc5855886cd067bb"
Nov 02 13:02:25.977080 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:25.977030    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027839d9e4d5412f3918697be611233ab8e597520e7205d4dc5855886cd067bb"} err="failed to get container status \"027839d9e4d5412f3918697be611233ab8e597520e7205d4dc5855886cd067bb\": rpc error: code = NotFound desc = could not find container \"027839d9e4d5412f3918697be611233ab8e597520e7205d4dc5855886cd067bb\": container with ID starting with 027839d9e4d5412f3918697be611233ab8e597520e7205d4dc5855886cd067bb not found: ID does not exist"
Nov 02 13:02:26.135612 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:26.135577    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"]
Nov 02 13:02:26.186296 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:26.186263    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-858chnm85"]
Nov 02 13:02:27.581889 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:27.581858    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11058bd3-db12-4168-9707-eb0431e3754d" path="/var/lib/kubelet/pods/11058bd3-db12-4168-9707-eb0431e3754d/volumes"
Nov 02 13:02:28.292630 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:02:28.292593    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:02:32.564464 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:32.564429    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:02:34.187900 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.187866    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"]
Nov 02 13:02:34.188406 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.188390    4464 cpu_manager.go:414] "containerMap: removing container" podUID="11058bd3-db12-4168-9707-eb0431e3754d" containerName="payload-processing"
Nov 02 13:02:34.188457 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.188408    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11058bd3-db12-4168-9707-eb0431e3754d" containerName="payload-processing"
Nov 02 13:02:34.188457 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.188428    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6a0fbb5a-0637-4743-ac99-e4d2fb34feb7" containerName="payload-processing"
Nov 02 13:02:34.188457 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.188433    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0fbb5a-0637-4743-ac99-e4d2fb34feb7" containerName="payload-processing"
Nov 02 13:02:34.188457 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.188441    4464 cpu_manager.go:414] "containerMap: removing container" podUID="4e37cbdd-16a5-4d30-865c-fee2a8d277a0" containerName="revoke-keys"
Nov 02 13:02:34.188457 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.188446    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e37cbdd-16a5-4d30-865c-fee2a8d277a0" containerName="revoke-keys"
Nov 02 13:02:34.188647 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.188526    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="11058bd3-db12-4168-9707-eb0431e3754d" containerName="payload-processing"
Nov 02 13:02:34.188647 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.188533    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0fbb5a-0637-4743-ac99-e4d2fb34feb7" containerName="payload-processing"
Nov 02 13:02:34.188647 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.188540    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e37cbdd-16a5-4d30-865c-fee2a8d277a0" containerName="revoke-keys"
Nov 02 13:02:34.193099 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.193082    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:02:34.250153 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.250128    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"]
Nov 02 13:02:34.298043 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.298018    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"routing-overlay\""
Nov 02 13:02:34.298043 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.298017    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3a78b4\""
Nov 02 13:02:34.298043 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.298047    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-5df25e9c8d4d\""
Nov 02 13:02:34.298283 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.298027    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"openshift-service-ca.crt\""
Nov 02 13:02:34.298283 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.298114    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"kube-root-ca.crt\""
Nov 02 13:02:34.298283 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.298161    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"payload-processing-external-model-e2e--a8ab13a3-dockercfg-b8t6r\""
Nov 02 13:02:34.369276 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.369252    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95108efb-2930-4c24-89f1-b68a3328264b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx\" (UID: \"95108efb-2930-4c24-89f1-b68a3328264b\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:02:34.369416 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.369289    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95108efb-2930-4c24-89f1-b68a3328264b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx\" (UID: \"95108efb-2930-4c24-89f1-b68a3328264b\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:02:34.369416 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.369322    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spbh9\" (UniqueName: \"kubernetes.io/projected/95108efb-2930-4c24-89f1-b68a3328264b-kube-api-access-spbh9\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx\" (UID: \"95108efb-2930-4c24-89f1-b68a3328264b\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:02:34.369505 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.369463    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95108efb-2930-4c24-89f1-b68a3328264b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx\" (UID: \"95108efb-2930-4c24-89f1-b68a3328264b\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:02:34.470226 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.470202    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95108efb-2930-4c24-89f1-b68a3328264b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx\" (UID: \"95108efb-2930-4c24-89f1-b68a3328264b\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:02:34.470372 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.470242    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95108efb-2930-4c24-89f1-b68a3328264b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx\" (UID: \"95108efb-2930-4c24-89f1-b68a3328264b\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:02:34.470372 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.470277    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spbh9\" (UniqueName: \"kubernetes.io/projected/95108efb-2930-4c24-89f1-b68a3328264b-kube-api-access-spbh9\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx\" (UID: \"95108efb-2930-4c24-89f1-b68a3328264b\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:02:34.470520 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.470386    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95108efb-2930-4c24-89f1-b68a3328264b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx\" (UID: \"95108efb-2930-4c24-89f1-b68a3328264b\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:02:34.470933 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.470912    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95108efb-2930-4c24-89f1-b68a3328264b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx\" (UID: \"95108efb-2930-4c24-89f1-b68a3328264b\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:02:34.471066 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.471048    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95108efb-2930-4c24-89f1-b68a3328264b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx\" (UID: \"95108efb-2930-4c24-89f1-b68a3328264b\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:02:34.472754 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.472735    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95108efb-2930-4c24-89f1-b68a3328264b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx\" (UID: \"95108efb-2930-4c24-89f1-b68a3328264b\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:02:34.532006 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.531961    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spbh9\" (UniqueName: \"kubernetes.io/projected/95108efb-2930-4c24-89f1-b68a3328264b-kube-api-access-spbh9\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx\" (UID: \"95108efb-2930-4c24-89f1-b68a3328264b\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:02:34.806056 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:34.805956    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:02:35.180856 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:35.180821    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"]
Nov 02 13:02:35.181860 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:02:35.181837    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95108efb_2930_4c24_89f1_b68a3328264b.slice/crio-c7f8a2e7a8f541a7efcc0203e87a9a11beabb0eae786e86ca0b10f1cbd66b1cd WatchSource:0}: Error finding container c7f8a2e7a8f541a7efcc0203e87a9a11beabb0eae786e86ca0b10f1cbd66b1cd: Status 404 returned error can't find the container with id c7f8a2e7a8f541a7efcc0203e87a9a11beabb0eae786e86ca0b10f1cbd66b1cd
Nov 02 13:02:36.023336 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:36.023296    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx" event={"ID":"95108efb-2930-4c24-89f1-b68a3328264b","Type":"ContainerStarted","Data":"5f0f6f504026db2af20ec329fcf364a795c5b6671ed0cd7ace73548c96ce27ab"}
Nov 02 13:02:36.023778 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:36.023353    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx" event={"ID":"95108efb-2930-4c24-89f1-b68a3328264b","Type":"ContainerStarted","Data":"c7f8a2e7a8f541a7efcc0203e87a9a11beabb0eae786e86ca0b10f1cbd66b1cd"}
Nov 02 13:02:36.023778 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:36.023375    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:02:36.142349 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:36.142300    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx" podStartSLOduration=2.142288162 podStartE2EDuration="2.142288162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:02:36.140712821 +0000 UTC m=+3984.704421232" watchObservedRunningTime="2026-11-02 13:02:36.142288162 +0000 UTC m=+3984.705996570"
Nov 02 13:02:36.829327 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:36.829290    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"]
Nov 02 13:02:36.833156 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:36.833138    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:02:36.938768 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:36.938738    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"agc-cred-0994b6920024\""
Nov 02 13:02:36.938950 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:36.938826    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6ac03f\""
Nov 02 13:02:36.938950 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:36.938829    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"kube-root-ca.crt\""
Nov 02 13:02:36.938950 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:36.938833    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"payload-processing-external-model-e2e--75a83f02-dockercfg-mzk4l\""
Nov 02 13:02:36.938950 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:36.938873    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"routing-overlay\""
Nov 02 13:02:36.939127 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:36.938961    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"openshift-service-ca.crt\""
Nov 02 13:02:36.941296 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:36.941273    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"]
Nov 02 13:02:36.998175 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:36.998148    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ed95fce-2f43-4783-8c0f-20b8702d6004-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg\" (UID: \"0ed95fce-2f43-4783-8c0f-20b8702d6004\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:02:36.998316 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:36.998203    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2zrk\" (UniqueName: \"kubernetes.io/projected/0ed95fce-2f43-4783-8c0f-20b8702d6004-kube-api-access-s2zrk\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg\" (UID: \"0ed95fce-2f43-4783-8c0f-20b8702d6004\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:02:36.998362 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:36.998323    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed95fce-2f43-4783-8c0f-20b8702d6004-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg\" (UID: \"0ed95fce-2f43-4783-8c0f-20b8702d6004\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:02:36.998362 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:36.998354    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ed95fce-2f43-4783-8c0f-20b8702d6004-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg\" (UID: \"0ed95fce-2f43-4783-8c0f-20b8702d6004\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:02:37.099156 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:37.099084    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ed95fce-2f43-4783-8c0f-20b8702d6004-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg\" (UID: \"0ed95fce-2f43-4783-8c0f-20b8702d6004\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:02:37.099156 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:37.099139    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2zrk\" (UniqueName: \"kubernetes.io/projected/0ed95fce-2f43-4783-8c0f-20b8702d6004-kube-api-access-s2zrk\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg\" (UID: \"0ed95fce-2f43-4783-8c0f-20b8702d6004\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:02:37.099656 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:37.099202    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed95fce-2f43-4783-8c0f-20b8702d6004-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg\" (UID: \"0ed95fce-2f43-4783-8c0f-20b8702d6004\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:02:37.099656 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:37.099219    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ed95fce-2f43-4783-8c0f-20b8702d6004-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg\" (UID: \"0ed95fce-2f43-4783-8c0f-20b8702d6004\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:02:37.099729 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:37.099708    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ed95fce-2f43-4783-8c0f-20b8702d6004-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg\" (UID: \"0ed95fce-2f43-4783-8c0f-20b8702d6004\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:02:37.099825 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:37.099805    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed95fce-2f43-4783-8c0f-20b8702d6004-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg\" (UID: \"0ed95fce-2f43-4783-8c0f-20b8702d6004\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:02:37.101537 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:37.101518    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ed95fce-2f43-4783-8c0f-20b8702d6004-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg\" (UID: \"0ed95fce-2f43-4783-8c0f-20b8702d6004\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:02:37.161591 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:37.161562    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2zrk\" (UniqueName: \"kubernetes.io/projected/0ed95fce-2f43-4783-8c0f-20b8702d6004-kube-api-access-s2zrk\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg\" (UID: \"0ed95fce-2f43-4783-8c0f-20b8702d6004\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:02:37.447155 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:37.447081    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:02:37.820384 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:37.820357    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"]
Nov 02 13:02:37.821345 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:02:37.821319    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed95fce_2f43_4783_8c0f_20b8702d6004.slice/crio-2a6b216212a5a74de8912333f19e8d6afc8016da8368e4f07cdea1953f2211e8 WatchSource:0}: Error finding container 2a6b216212a5a74de8912333f19e8d6afc8016da8368e4f07cdea1953f2211e8: Status 404 returned error can't find the container with id 2a6b216212a5a74de8912333f19e8d6afc8016da8368e4f07cdea1953f2211e8
Nov 02 13:02:38.035333 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:38.035301    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg" event={"ID":"0ed95fce-2f43-4783-8c0f-20b8702d6004","Type":"ContainerStarted","Data":"e5827bcdceca6344e49104f158f2ade2eb8e0b25df32c4e39c486f0507828208"}
Nov 02 13:02:38.035482 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:38.035350    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg" event={"ID":"0ed95fce-2f43-4783-8c0f-20b8702d6004","Type":"ContainerStarted","Data":"2a6b216212a5a74de8912333f19e8d6afc8016da8368e4f07cdea1953f2211e8"}
Nov 02 13:02:38.035482 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:38.035406    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:02:38.155746 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:38.155649    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg" podStartSLOduration=2.155635151 podStartE2EDuration="2.155635151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:02:38.15344331 +0000 UTC m=+3986.717151753" watchObservedRunningTime="2026-11-02 13:02:38.155635151 +0000 UTC m=+3986.719343559"
Nov 02 13:02:47.031628 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:47.031597    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:02:49.043734 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:02:49.043698    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:02:58.294459 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:02:58.294359    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:03:05.678434 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:05.678399    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:03:28.292588 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:03:28.292542    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:03:33.071251 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:33.071212    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"]
Nov 02 13:03:33.122216 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:33.122183    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05b22116-dnpd7"]
Nov 02 13:03:33.581883 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:33.581840    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4874f4e9-22d3-40ba-bb62-1eb48865dd05" path="/var/lib/kubelet/pods/4874f4e9-22d3-40ba-bb62-1eb48865dd05/volumes"
Nov 02 13:03:38.474352 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:38.474315    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:03:46.048516 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:46.048478    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"]
Nov 02 13:03:46.051523 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:46.051495    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8257f59d-gm9kj"]
Nov 02 13:03:47.293382 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.293346    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"]
Nov 02 13:03:47.297682 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.297663    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:03:47.352480 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.352436    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-2c377fbf861a\""
Nov 02 13:03:47.405903 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.405871    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"]
Nov 02 13:03:47.414266 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.414244    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69f44410-ee12-4afc-96fb-073cdfd73b73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n\" (UID: \"69f44410-ee12-4afc-96fb-073cdfd73b73\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:03:47.414387 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.414308    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69f44410-ee12-4afc-96fb-073cdfd73b73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n\" (UID: \"69f44410-ee12-4afc-96fb-073cdfd73b73\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:03:47.414449 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.414409    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fzwm\" (UniqueName: \"kubernetes.io/projected/69f44410-ee12-4afc-96fb-073cdfd73b73-kube-api-access-7fzwm\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n\" (UID: \"69f44410-ee12-4afc-96fb-073cdfd73b73\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:03:47.414629 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.414600    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f44410-ee12-4afc-96fb-073cdfd73b73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n\" (UID: \"69f44410-ee12-4afc-96fb-073cdfd73b73\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:03:47.515878 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.515837    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fzwm\" (UniqueName: \"kubernetes.io/projected/69f44410-ee12-4afc-96fb-073cdfd73b73-kube-api-access-7fzwm\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n\" (UID: \"69f44410-ee12-4afc-96fb-073cdfd73b73\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:03:47.516106 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.515995    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f44410-ee12-4afc-96fb-073cdfd73b73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n\" (UID: \"69f44410-ee12-4afc-96fb-073cdfd73b73\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:03:47.516106 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.516038    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69f44410-ee12-4afc-96fb-073cdfd73b73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n\" (UID: \"69f44410-ee12-4afc-96fb-073cdfd73b73\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:03:47.516257 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.516117    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69f44410-ee12-4afc-96fb-073cdfd73b73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n\" (UID: \"69f44410-ee12-4afc-96fb-073cdfd73b73\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:03:47.516799 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.516775    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69f44410-ee12-4afc-96fb-073cdfd73b73-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n\" (UID: \"69f44410-ee12-4afc-96fb-073cdfd73b73\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:03:47.516960 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.516803    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f44410-ee12-4afc-96fb-073cdfd73b73-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n\" (UID: \"69f44410-ee12-4afc-96fb-073cdfd73b73\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:03:47.518519 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.518500    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69f44410-ee12-4afc-96fb-073cdfd73b73-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n\" (UID: \"69f44410-ee12-4afc-96fb-073cdfd73b73\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:03:47.578337 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.578264    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fzwm\" (UniqueName: \"kubernetes.io/projected/69f44410-ee12-4afc-96fb-073cdfd73b73-kube-api-access-7fzwm\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n\" (UID: \"69f44410-ee12-4afc-96fb-073cdfd73b73\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:03:47.582020 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.581993    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a357fb-c9b7-412e-8b29-3ba6cb4718f9" path="/var/lib/kubelet/pods/d8a357fb-c9b7-412e-8b29-3ba6cb4718f9/volumes"
Nov 02 13:03:47.610969 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.610945    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:03:47.987787 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:47.986476    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"]
Nov 02 13:03:47.988752 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:03:47.988705    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f44410_ee12_4afc_96fb_073cdfd73b73.slice/crio-5d77843959ae89ef8c250b7bde2a7e4d8590b061376ec72fc05672f89b58599b WatchSource:0}: Error finding container 5d77843959ae89ef8c250b7bde2a7e4d8590b061376ec72fc05672f89b58599b: Status 404 returned error can't find the container with id 5d77843959ae89ef8c250b7bde2a7e4d8590b061376ec72fc05672f89b58599b
Nov 02 13:03:48.430957 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:48.430856    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n" event={"ID":"69f44410-ee12-4afc-96fb-073cdfd73b73","Type":"ContainerStarted","Data":"fa8436d5e1f22aedb6ab38e1b09198cba75251677e6dd3e69338c343b0892129"}
Nov 02 13:03:48.430957 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:48.430945    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n" event={"ID":"69f44410-ee12-4afc-96fb-073cdfd73b73","Type":"ContainerStarted","Data":"5d77843959ae89ef8c250b7bde2a7e4d8590b061376ec72fc05672f89b58599b"}
Nov 02 13:03:48.431502 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:48.431056    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:03:48.551103 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:48.551047    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n" podStartSLOduration=1.551033681 podStartE2EDuration="1.551033681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:03:48.549745534 +0000 UTC m=+4057.113453944" watchObservedRunningTime="2026-11-02 13:03:48.551033681 +0000 UTC m=+4057.114742106"
Nov 02 13:03:49.895776 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:49.895735    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"]
Nov 02 13:03:49.899900 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:49.899877    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:03:49.954940 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:49.954902    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"agc-cred-add3ebb37edc\""
Nov 02 13:03:49.957752 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:49.957726    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"]
Nov 02 13:03:50.043464 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:50.043417    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62fp\" (UniqueName: \"kubernetes.io/projected/3dbaad92-d33e-4573-bda3-b5fa373e60e1-kube-api-access-s62fp\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz\" (UID: \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:03:50.043464 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:50.043463    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3dbaad92-d33e-4573-bda3-b5fa373e60e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz\" (UID: \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:03:50.043734 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:50.043491    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3dbaad92-d33e-4573-bda3-b5fa373e60e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz\" (UID: \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:03:50.043734 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:50.043626    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dbaad92-d33e-4573-bda3-b5fa373e60e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz\" (UID: \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:03:50.145016 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:50.144943    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62fp\" (UniqueName: \"kubernetes.io/projected/3dbaad92-d33e-4573-bda3-b5fa373e60e1-kube-api-access-s62fp\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz\" (UID: \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:03:50.145208 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:50.145030    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3dbaad92-d33e-4573-bda3-b5fa373e60e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz\" (UID: \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:03:50.145208 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:50.145054    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3dbaad92-d33e-4573-bda3-b5fa373e60e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz\" (UID: \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:03:50.145208 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:50.145106    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dbaad92-d33e-4573-bda3-b5fa373e60e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz\" (UID: \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:03:50.145899 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:50.145851    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3dbaad92-d33e-4573-bda3-b5fa373e60e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz\" (UID: \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:03:50.145961 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:50.145947    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dbaad92-d33e-4573-bda3-b5fa373e60e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz\" (UID: \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:03:50.147582 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:50.147564    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3dbaad92-d33e-4573-bda3-b5fa373e60e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz\" (UID: \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:03:50.207185 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:50.207152    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62fp\" (UniqueName: \"kubernetes.io/projected/3dbaad92-d33e-4573-bda3-b5fa373e60e1-kube-api-access-s62fp\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz\" (UID: \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:03:50.213932 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:50.213914    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:03:50.587026 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:50.587000    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"]
Nov 02 13:03:50.588324 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:03:50.588294    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbaad92_d33e_4573_bda3_b5fa373e60e1.slice/crio-1e1dc87611ff982814a634846d49821a8a3ccacb44a9e7b234297dd8a37fe784 WatchSource:0}: Error finding container 1e1dc87611ff982814a634846d49821a8a3ccacb44a9e7b234297dd8a37fe784: Status 404 returned error can't find the container with id 1e1dc87611ff982814a634846d49821a8a3ccacb44a9e7b234297dd8a37fe784
Nov 02 13:03:51.450071 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:51.450027    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz" event={"ID":"3dbaad92-d33e-4573-bda3-b5fa373e60e1","Type":"ContainerStarted","Data":"c7e02363915d6321f0510ce08deb32896aca9553f01575cf6fa8c55d8bd7cdee"}
Nov 02 13:03:51.450470 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:51.450086    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz" event={"ID":"3dbaad92-d33e-4573-bda3-b5fa373e60e1","Type":"ContainerStarted","Data":"1e1dc87611ff982814a634846d49821a8a3ccacb44a9e7b234297dd8a37fe784"}
Nov 02 13:03:51.450470 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:51.450113    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:03:51.568772 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:51.568715    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz" podStartSLOduration=2.568703045 podStartE2EDuration="2.568703045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:03:51.568282637 +0000 UTC m=+4060.131991043" watchObservedRunningTime="2026-11-02 13:03:51.568703045 +0000 UTC m=+4060.132411454"
Nov 02 13:03:52.578251 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:52.578217    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:03:58.048503 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:58.048464    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"]
Nov 02 13:03:58.099187 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:58.099161    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-524e6df4-l2jm8"]
Nov 02 13:03:58.291103 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:03:58.291064    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:03:59.440259 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:59.440234    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:03:59.583811 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:59.583767    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d9bbc60-8d04-45d8-a35a-c90392f41bb1" path="/var/lib/kubelet/pods/8d9bbc60-8d04-45d8-a35a-c90392f41bb1/volumes"
Nov 02 13:03:59.614525 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:03:59.614483    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"]
Nov 02 13:04:02.459169 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:02.459142    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:04:02.633470 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:02.633437    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"]
Nov 02 13:04:04.615793 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:04.615753    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx" podUID="95108efb-2930-4c24-89f1-b68a3328264b" containerName="payload-processing" containerID="cri-o://5f0f6f504026db2af20ec329fcf364a795c5b6671ed0cd7ace73548c96ce27ab" gracePeriod=25
Nov 02 13:04:04.874477 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:04.874425    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:04:04.919604 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:04.919568    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95108efb-2930-4c24-89f1-b68a3328264b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/95108efb-2930-4c24-89f1-b68a3328264b-provider-credentials\") pod \"95108efb-2930-4c24-89f1-b68a3328264b\" (UID: \"95108efb-2930-4c24-89f1-b68a3328264b\") "
Nov 02 13:04:04.919751 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:04.919624    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95108efb-2930-4c24-89f1-b68a3328264b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95108efb-2930-4c24-89f1-b68a3328264b-plugins-config-volume\") pod \"95108efb-2930-4c24-89f1-b68a3328264b\" (UID: \"95108efb-2930-4c24-89f1-b68a3328264b\") "
Nov 02 13:04:04.919751 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:04.919722    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95108efb-2930-4c24-89f1-b68a3328264b-kube-api-access-spbh9\" (UniqueName: \"kubernetes.io/projected/95108efb-2930-4c24-89f1-b68a3328264b-kube-api-access-spbh9\") pod \"95108efb-2930-4c24-89f1-b68a3328264b\" (UID: \"95108efb-2930-4c24-89f1-b68a3328264b\") "
Nov 02 13:04:04.919827 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:04.919761    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95108efb-2930-4c24-89f1-b68a3328264b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95108efb-2930-4c24-89f1-b68a3328264b-routing-overlay-volume\") pod \"95108efb-2930-4c24-89f1-b68a3328264b\" (UID: \"95108efb-2930-4c24-89f1-b68a3328264b\") "
Nov 02 13:04:04.920178 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:04.920147    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95108efb-2930-4c24-89f1-b68a3328264b-plugins-config-volume" pod "95108efb-2930-4c24-89f1-b68a3328264b" (UID: "95108efb-2930-4c24-89f1-b68a3328264b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:04:04.920178 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:04.920166    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95108efb-2930-4c24-89f1-b68a3328264b-routing-overlay-volume" pod "95108efb-2930-4c24-89f1-b68a3328264b" (UID: "95108efb-2930-4c24-89f1-b68a3328264b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:04:04.921559 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:04.921537    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95108efb-2930-4c24-89f1-b68a3328264b-provider-credentials" pod "95108efb-2930-4c24-89f1-b68a3328264b" (UID: "95108efb-2930-4c24-89f1-b68a3328264b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:04:04.921811 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:04.921788    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95108efb-2930-4c24-89f1-b68a3328264b-kube-api-access-spbh9" pod "95108efb-2930-4c24-89f1-b68a3328264b" (UID: "95108efb-2930-4c24-89f1-b68a3328264b"). InnerVolumeSpecName "kube-api-access-spbh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:04:05.021345 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:05.021307    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spbh9\" (UniqueName: \"kubernetes.io/projected/95108efb-2930-4c24-89f1-b68a3328264b-kube-api-access-spbh9\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:04:05.021345 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:05.021340    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95108efb-2930-4c24-89f1-b68a3328264b-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:04:05.021553 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:05.021353    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95108efb-2930-4c24-89f1-b68a3328264b-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:04:05.021553 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:05.021367    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95108efb-2930-4c24-89f1-b68a3328264b-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:04:05.531952 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:05.531915    4464 generic.go:356] "Generic (PLEG): container finished" podID="95108efb-2930-4c24-89f1-b68a3328264b" containerID="5f0f6f504026db2af20ec329fcf364a795c5b6671ed0cd7ace73548c96ce27ab" exitCode=0
Nov 02 13:04:05.532148 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:05.531966    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"
Nov 02 13:04:05.532148 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:05.532005    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx" event={"ID":"95108efb-2930-4c24-89f1-b68a3328264b","Type":"ContainerDied","Data":"5f0f6f504026db2af20ec329fcf364a795c5b6671ed0cd7ace73548c96ce27ab"}
Nov 02 13:04:05.532148 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:05.532064    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx" event={"ID":"95108efb-2930-4c24-89f1-b68a3328264b","Type":"ContainerDied","Data":"c7f8a2e7a8f541a7efcc0203e87a9a11beabb0eae786e86ca0b10f1cbd66b1cd"}
Nov 02 13:04:05.532148 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:05.532086    4464 scope.go:122] "RemoveContainer" containerID="5f0f6f504026db2af20ec329fcf364a795c5b6671ed0cd7ace73548c96ce27ab"
Nov 02 13:04:05.542339 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:05.542324    4464 scope.go:122] "RemoveContainer" containerID="5f0f6f504026db2af20ec329fcf364a795c5b6671ed0cd7ace73548c96ce27ab"
Nov 02 13:04:05.542604 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:04:05.542587    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f0f6f504026db2af20ec329fcf364a795c5b6671ed0cd7ace73548c96ce27ab\": container with ID starting with 5f0f6f504026db2af20ec329fcf364a795c5b6671ed0cd7ace73548c96ce27ab not found: ID does not exist" containerID="5f0f6f504026db2af20ec329fcf364a795c5b6671ed0cd7ace73548c96ce27ab"
Nov 02 13:04:05.542656 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:05.542613    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0f6f504026db2af20ec329fcf364a795c5b6671ed0cd7ace73548c96ce27ab"} err="failed to get container status \"5f0f6f504026db2af20ec329fcf364a795c5b6671ed0cd7ace73548c96ce27ab\": rpc error: code = NotFound desc = could not find container \"5f0f6f504026db2af20ec329fcf364a795c5b6671ed0cd7ace73548c96ce27ab\": container with ID starting with 5f0f6f504026db2af20ec329fcf364a795c5b6671ed0cd7ace73548c96ce27ab not found: ID does not exist"
Nov 02 13:04:05.698761 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:05.698729    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"]
Nov 02 13:04:05.749223 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:05.749190    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-868887ndx"]
Nov 02 13:04:06.915168 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:06.915131    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:04:07.580966 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:07.580934    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95108efb-2930-4c24-89f1-b68a3328264b" path="/var/lib/kubelet/pods/95108efb-2930-4c24-89f1-b68a3328264b/volumes"
Nov 02 13:04:07.634228 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:07.634194    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg" podUID="0ed95fce-2f43-4783-8c0f-20b8702d6004" containerName="payload-processing" containerID="cri-o://e5827bcdceca6344e49104f158f2ade2eb8e0b25df32c4e39c486f0507828208" gracePeriod=25
Nov 02 13:04:07.887759 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:07.887736    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:04:07.957876 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:07.957844    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed95fce-2f43-4783-8c0f-20b8702d6004-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ed95fce-2f43-4783-8c0f-20b8702d6004-routing-overlay-volume\") pod \"0ed95fce-2f43-4783-8c0f-20b8702d6004\" (UID: \"0ed95fce-2f43-4783-8c0f-20b8702d6004\") "
Nov 02 13:04:07.958291 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:07.957930    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed95fce-2f43-4783-8c0f-20b8702d6004-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed95fce-2f43-4783-8c0f-20b8702d6004-plugins-config-volume\") pod \"0ed95fce-2f43-4783-8c0f-20b8702d6004\" (UID: \"0ed95fce-2f43-4783-8c0f-20b8702d6004\") "
Nov 02 13:04:07.958291 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:07.958044    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed95fce-2f43-4783-8c0f-20b8702d6004-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ed95fce-2f43-4783-8c0f-20b8702d6004-provider-credentials\") pod \"0ed95fce-2f43-4783-8c0f-20b8702d6004\" (UID: \"0ed95fce-2f43-4783-8c0f-20b8702d6004\") "
Nov 02 13:04:07.958291 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:07.958077    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed95fce-2f43-4783-8c0f-20b8702d6004-kube-api-access-s2zrk\" (UniqueName: \"kubernetes.io/projected/0ed95fce-2f43-4783-8c0f-20b8702d6004-kube-api-access-s2zrk\") pod \"0ed95fce-2f43-4783-8c0f-20b8702d6004\" (UID: \"0ed95fce-2f43-4783-8c0f-20b8702d6004\") "
Nov 02 13:04:07.958291 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:07.958267    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed95fce-2f43-4783-8c0f-20b8702d6004-routing-overlay-volume" pod "0ed95fce-2f43-4783-8c0f-20b8702d6004" (UID: "0ed95fce-2f43-4783-8c0f-20b8702d6004"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:04:07.958515 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:07.958346    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed95fce-2f43-4783-8c0f-20b8702d6004-plugins-config-volume" pod "0ed95fce-2f43-4783-8c0f-20b8702d6004" (UID: "0ed95fce-2f43-4783-8c0f-20b8702d6004"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:04:07.958582 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:07.958564    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ed95fce-2f43-4783-8c0f-20b8702d6004-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:04:07.958624 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:07.958586    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ed95fce-2f43-4783-8c0f-20b8702d6004-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:04:07.960418 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:07.960396    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed95fce-2f43-4783-8c0f-20b8702d6004-kube-api-access-s2zrk" pod "0ed95fce-2f43-4783-8c0f-20b8702d6004" (UID: "0ed95fce-2f43-4783-8c0f-20b8702d6004"). InnerVolumeSpecName "kube-api-access-s2zrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:04:07.961554 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:07.961532    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed95fce-2f43-4783-8c0f-20b8702d6004-provider-credentials" pod "0ed95fce-2f43-4783-8c0f-20b8702d6004" (UID: "0ed95fce-2f43-4783-8c0f-20b8702d6004"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:04:08.060161 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:08.060126    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ed95fce-2f43-4783-8c0f-20b8702d6004-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:04:08.060161 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:08.060155    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2zrk\" (UniqueName: \"kubernetes.io/projected/0ed95fce-2f43-4783-8c0f-20b8702d6004-kube-api-access-s2zrk\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:04:08.558370 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:08.558334    4464 generic.go:356] "Generic (PLEG): container finished" podID="0ed95fce-2f43-4783-8c0f-20b8702d6004" containerID="e5827bcdceca6344e49104f158f2ade2eb8e0b25df32c4e39c486f0507828208" exitCode=0
Nov 02 13:04:08.558557 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:08.558389    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"
Nov 02 13:04:08.558557 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:08.558400    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg" event={"ID":"0ed95fce-2f43-4783-8c0f-20b8702d6004","Type":"ContainerDied","Data":"e5827bcdceca6344e49104f158f2ade2eb8e0b25df32c4e39c486f0507828208"}
Nov 02 13:04:08.558557 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:08.558447    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg" event={"ID":"0ed95fce-2f43-4783-8c0f-20b8702d6004","Type":"ContainerDied","Data":"2a6b216212a5a74de8912333f19e8d6afc8016da8368e4f07cdea1953f2211e8"}
Nov 02 13:04:08.558557 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:08.558469    4464 scope.go:122] "RemoveContainer" containerID="e5827bcdceca6344e49104f158f2ade2eb8e0b25df32c4e39c486f0507828208"
Nov 02 13:04:08.567876 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:08.567860    4464 scope.go:122] "RemoveContainer" containerID="e5827bcdceca6344e49104f158f2ade2eb8e0b25df32c4e39c486f0507828208"
Nov 02 13:04:08.568160 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:04:08.568142    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5827bcdceca6344e49104f158f2ade2eb8e0b25df32c4e39c486f0507828208\": container with ID starting with e5827bcdceca6344e49104f158f2ade2eb8e0b25df32c4e39c486f0507828208 not found: ID does not exist" containerID="e5827bcdceca6344e49104f158f2ade2eb8e0b25df32c4e39c486f0507828208"
Nov 02 13:04:08.568204 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:08.568168    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5827bcdceca6344e49104f158f2ade2eb8e0b25df32c4e39c486f0507828208"} err="failed to get container status \"e5827bcdceca6344e49104f158f2ade2eb8e0b25df32c4e39c486f0507828208\": rpc error: code = NotFound desc = could not find container \"e5827bcdceca6344e49104f158f2ade2eb8e0b25df32c4e39c486f0507828208\": container with ID starting with e5827bcdceca6344e49104f158f2ade2eb8e0b25df32c4e39c486f0507828208 not found: ID does not exist"
Nov 02 13:04:08.725107 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:08.725074    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"]
Nov 02 13:04:08.775671 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:08.775636    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655zdhtg"]
Nov 02 13:04:09.581335 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:09.581297    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed95fce-2f43-4783-8c0f-20b8702d6004" path="/var/lib/kubelet/pods/0ed95fce-2f43-4783-8c0f-20b8702d6004/volumes"
Nov 02 13:04:14.047204 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:14.047165    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"]
Nov 02 13:04:14.097705 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:14.097674    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edcd0990-n7b7m"]
Nov 02 13:04:15.581525 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:15.581495    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e32b65-70ea-42cd-aa32-dbd4bd704fb9" path="/var/lib/kubelet/pods/40e32b65-70ea-42cd-aa32-dbd4bd704fb9/volumes"
Nov 02 13:04:18.801478 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:04:18.801440    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:04:28.291986 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:04:28.291898    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:04:58.291667 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:04:58.291625    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:05:14.314840 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:05:14.314811    4464 scope.go:122] "RemoveContainer" containerID="f5a033c18ae532cf891832d54711982c49a61bef8967abfdf1e580677a238d49"
Nov 02 13:05:14.323835 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:05:14.323815    4464 scope.go:122] "RemoveContainer" containerID="2f910a44ba251989a92e9ca4b373bf45d2ddc7911c3fa2efa2fa53b10d7556d6"
Nov 02 13:05:14.332325 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:05:14.332298    4464 scope.go:122] "RemoveContainer" containerID="ae8b6a0ce0653b038c91e158abdf370e868aa9b72cd6cf2b3cf91e04bcdf3e5a"
Nov 02 13:05:14.340012 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:05:14.339952    4464 scope.go:122] "RemoveContainer" containerID="ffa0e1b0799f88a469ed5b68977495acab1daf991525974c1309388d3e1d9c9b"
Nov 02 13:05:22.050701 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:05:22.050660    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"]
Nov 02 13:05:22.101139 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:05:22.101103    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564d99db-mk457"]
Nov 02 13:05:23.581237 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:05:23.581205    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be60869f-1203-4f0f-aa25-1d6973eef5ef" path="/var/lib/kubelet/pods/be60869f-1203-4f0f-aa25-1d6973eef5ef/volumes"
Nov 02 13:05:28.291831 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:05:28.291795    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:05:58.291843 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:05:58.291770    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:06:12.129709 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:12.129681    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:06:12.146280 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:12.146259    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:06:12.173062 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:12.173041    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:06:12.189450 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:12.189431    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:06:21.927685 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:21.927649    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"]
Nov 02 13:06:21.928162 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:21.927859    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm" podUID="2e052c7c-9a9d-482d-8757-1b160b965466" containerName="istio-proxy" containerID="cri-o://7484deb6afd0d18571bdefcdfca60aad0938a26695848e024085b13c1900fa26" gracePeriod=30
Nov 02 13:06:27.186937 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.186909    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:06:27.250162 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.250136    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-workload-socket\") pod \"2e052c7c-9a9d-482d-8757-1b160b965466\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") "
Nov 02 13:06:27.250358 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.250173    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-istio-envoy\") pod \"2e052c7c-9a9d-482d-8757-1b160b965466\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") "
Nov 02 13:06:27.250358 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.250194    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-workload-certs\") pod \"2e052c7c-9a9d-482d-8757-1b160b965466\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") "
Nov 02 13:06:27.250358 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.250242    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e052c7c-9a9d-482d-8757-1b160b965466-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e052c7c-9a9d-482d-8757-1b160b965466-istio-podinfo\") pod \"2e052c7c-9a9d-482d-8757-1b160b965466\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") "
Nov 02 13:06:27.250358 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.250264    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-credential-socket\") pod \"2e052c7c-9a9d-482d-8757-1b160b965466\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") "
Nov 02 13:06:27.250358 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.250332    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e052c7c-9a9d-482d-8757-1b160b965466-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e052c7c-9a9d-482d-8757-1b160b965466-istiod-ca-cert\") pod \"2e052c7c-9a9d-482d-8757-1b160b965466\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") "
Nov 02 13:06:27.250655 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.250377    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-istio-data\") pod \"2e052c7c-9a9d-482d-8757-1b160b965466\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") "
Nov 02 13:06:27.250655 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.250490    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-workload-socket" pod "2e052c7c-9a9d-482d-8757-1b160b965466" (UID: "2e052c7c-9a9d-482d-8757-1b160b965466"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:06:27.250655 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.250514    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e052c7c-9a9d-482d-8757-1b160b965466-kube-api-access-j56xq\" (UniqueName: \"kubernetes.io/projected/2e052c7c-9a9d-482d-8757-1b160b965466-kube-api-access-j56xq\") pod \"2e052c7c-9a9d-482d-8757-1b160b965466\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") "
Nov 02 13:06:27.250655 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.250571    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-workload-certs" pod "2e052c7c-9a9d-482d-8757-1b160b965466" (UID: "2e052c7c-9a9d-482d-8757-1b160b965466"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:06:27.250655 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.250612    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e052c7c-9a9d-482d-8757-1b160b965466-istio-token\" (UniqueName: \"kubernetes.io/projected/2e052c7c-9a9d-482d-8757-1b160b965466-istio-token\") pod \"2e052c7c-9a9d-482d-8757-1b160b965466\" (UID: \"2e052c7c-9a9d-482d-8757-1b160b965466\") "
Nov 02 13:06:27.250891 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.250667    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-istio-data" pod "2e052c7c-9a9d-482d-8757-1b160b965466" (UID: "2e052c7c-9a9d-482d-8757-1b160b965466"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:06:27.250891 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.250719    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-credential-socket" pod "2e052c7c-9a9d-482d-8757-1b160b965466" (UID: "2e052c7c-9a9d-482d-8757-1b160b965466"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:06:27.250891 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.250765    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e052c7c-9a9d-482d-8757-1b160b965466-istiod-ca-cert" pod "2e052c7c-9a9d-482d-8757-1b160b965466" (UID: "2e052c7c-9a9d-482d-8757-1b160b965466"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:27.251098 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.251064    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:27.251098 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.251086    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:27.251217 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.251100    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:27.251217 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.251113    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e052c7c-9a9d-482d-8757-1b160b965466-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:27.251217 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.251123    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:27.252832 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.252810    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e052c7c-9a9d-482d-8757-1b160b965466-kube-api-access-j56xq" pod "2e052c7c-9a9d-482d-8757-1b160b965466" (UID: "2e052c7c-9a9d-482d-8757-1b160b965466"). InnerVolumeSpecName "kube-api-access-j56xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:27.252955 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.252838    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e052c7c-9a9d-482d-8757-1b160b965466-istio-token" pod "2e052c7c-9a9d-482d-8757-1b160b965466" (UID: "2e052c7c-9a9d-482d-8757-1b160b965466"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:27.253022 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.252959    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-istio-envoy" pod "2e052c7c-9a9d-482d-8757-1b160b965466" (UID: "2e052c7c-9a9d-482d-8757-1b160b965466"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:06:27.253069 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.253046    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e052c7c-9a9d-482d-8757-1b160b965466-istio-podinfo" pod "2e052c7c-9a9d-482d-8757-1b160b965466" (UID: "2e052c7c-9a9d-482d-8757-1b160b965466"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:06:27.296235 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.296203    4464 generic.go:356] "Generic (PLEG): container finished" podID="2e052c7c-9a9d-482d-8757-1b160b965466" containerID="7484deb6afd0d18571bdefcdfca60aad0938a26695848e024085b13c1900fa26" exitCode=0
Nov 02 13:06:27.296382 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.296258    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"
Nov 02 13:06:27.296382 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.296292    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm" event={"ID":"2e052c7c-9a9d-482d-8757-1b160b965466","Type":"ContainerDied","Data":"7484deb6afd0d18571bdefcdfca60aad0938a26695848e024085b13c1900fa26"}
Nov 02 13:06:27.296382 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.296356    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm" event={"ID":"2e052c7c-9a9d-482d-8757-1b160b965466","Type":"ContainerDied","Data":"842be26033626c7dc85ea4f13cdc57c6be2d7f172d61cfb5bb5dc5e9b3445c7f"}
Nov 02 13:06:27.296562 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.296385    4464 scope.go:122] "RemoveContainer" containerID="7484deb6afd0d18571bdefcdfca60aad0938a26695848e024085b13c1900fa26"
Nov 02 13:06:27.307217 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.307197    4464 scope.go:122] "RemoveContainer" containerID="7484deb6afd0d18571bdefcdfca60aad0938a26695848e024085b13c1900fa26"
Nov 02 13:06:27.307497 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:06:27.307480    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7484deb6afd0d18571bdefcdfca60aad0938a26695848e024085b13c1900fa26\": container with ID starting with 7484deb6afd0d18571bdefcdfca60aad0938a26695848e024085b13c1900fa26 not found: ID does not exist" containerID="7484deb6afd0d18571bdefcdfca60aad0938a26695848e024085b13c1900fa26"
Nov 02 13:06:27.307574 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.307507    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7484deb6afd0d18571bdefcdfca60aad0938a26695848e024085b13c1900fa26"} err="failed to get container status \"7484deb6afd0d18571bdefcdfca60aad0938a26695848e024085b13c1900fa26\": rpc error: code = NotFound desc = could not find container \"7484deb6afd0d18571bdefcdfca60aad0938a26695848e024085b13c1900fa26\": container with ID starting with 7484deb6afd0d18571bdefcdfca60aad0938a26695848e024085b13c1900fa26 not found: ID does not exist"
Nov 02 13:06:27.352200 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.352174    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j56xq\" (UniqueName: \"kubernetes.io/projected/2e052c7c-9a9d-482d-8757-1b160b965466-kube-api-access-j56xq\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:27.352200 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.352198    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e052c7c-9a9d-482d-8757-1b160b965466-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:27.352404 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.352211    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e052c7c-9a9d-482d-8757-1b160b965466-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:27.352404 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.352220    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e052c7c-9a9d-482d-8757-1b160b965466-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:27.476078 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.476040    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"]
Nov 02 13:06:27.481386 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.481350    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-xvknm"]
Nov 02 13:06:27.581561 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:27.581475    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e052c7c-9a9d-482d-8757-1b160b965466" path="/var/lib/kubelet/pods/2e052c7c-9a9d-482d-8757-1b160b965466/volumes"
Nov 02 13:06:28.291177 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:06:28.291143    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:06:37.046135 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:37.046101    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"]
Nov 02 13:06:37.046719 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:37.046306    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt" podUID="b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" containerName="istio-proxy" containerID="cri-o://0d4a1aa1e9cd5b3562ad3d58571f31758fa096a7febb0c84e16327056d247e38" gracePeriod=30
Nov 02 13:06:37.097147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:37.097112    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"]
Nov 02 13:06:37.097396 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:37.097353    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r" podUID="f6025f4f-a2b0-4537-8a34-439e39b04658" containerName="istio-proxy" containerID="cri-o://a0a25fae4b32208e0c482fddced2b12d1ccd4ff505bc03b6556a0f10bee4c6e1" gracePeriod=30
Nov 02 13:06:42.342659 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.342618    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:06:42.375918 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.375890    4464 generic.go:356] "Generic (PLEG): container finished" podID="f6025f4f-a2b0-4537-8a34-439e39b04658" containerID="a0a25fae4b32208e0c482fddced2b12d1ccd4ff505bc03b6556a0f10bee4c6e1" exitCode=0
Nov 02 13:06:42.376058 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.375961    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r" event={"ID":"f6025f4f-a2b0-4537-8a34-439e39b04658","Type":"ContainerDied","Data":"a0a25fae4b32208e0c482fddced2b12d1ccd4ff505bc03b6556a0f10bee4c6e1"}
Nov 02 13:06:42.376058 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.376038    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r" event={"ID":"f6025f4f-a2b0-4537-8a34-439e39b04658","Type":"ContainerDied","Data":"72ead62ac3e3e41dc6463d7b909dfe9facb2b40be5fd70630e4f83eb4975fd9b"}
Nov 02 13:06:42.376058 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.376056    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72ead62ac3e3e41dc6463d7b909dfe9facb2b40be5fd70630e4f83eb4975fd9b"
Nov 02 13:06:42.377254 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.377232    4464 generic.go:356] "Generic (PLEG): container finished" podID="b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" containerID="0d4a1aa1e9cd5b3562ad3d58571f31758fa096a7febb0c84e16327056d247e38" exitCode=0
Nov 02 13:06:42.377355 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.377260    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt" event={"ID":"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51","Type":"ContainerDied","Data":"0d4a1aa1e9cd5b3562ad3d58571f31758fa096a7febb0c84e16327056d247e38"}
Nov 02 13:06:42.377355 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.377274    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"
Nov 02 13:06:42.377355 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.377289    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt" event={"ID":"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51","Type":"ContainerDied","Data":"8838b76271bcdc0dbd15c53c8c93c369785e28b34028d669b1391ff039773b32"}
Nov 02 13:06:42.377355 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.377316    4464 scope.go:122] "RemoveContainer" containerID="0d4a1aa1e9cd5b3562ad3d58571f31758fa096a7febb0c84e16327056d247e38"
Nov 02 13:06:42.378116 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.378099    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:06:42.385984 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.385955    4464 scope.go:122] "RemoveContainer" containerID="0d4a1aa1e9cd5b3562ad3d58571f31758fa096a7febb0c84e16327056d247e38"
Nov 02 13:06:42.386292 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:06:42.386270    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4a1aa1e9cd5b3562ad3d58571f31758fa096a7febb0c84e16327056d247e38\": container with ID starting with 0d4a1aa1e9cd5b3562ad3d58571f31758fa096a7febb0c84e16327056d247e38 not found: ID does not exist" containerID="0d4a1aa1e9cd5b3562ad3d58571f31758fa096a7febb0c84e16327056d247e38"
Nov 02 13:06:42.386383 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.386297    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4a1aa1e9cd5b3562ad3d58571f31758fa096a7febb0c84e16327056d247e38"} err="failed to get container status \"0d4a1aa1e9cd5b3562ad3d58571f31758fa096a7febb0c84e16327056d247e38\": rpc error: code = NotFound desc = could not find container \"0d4a1aa1e9cd5b3562ad3d58571f31758fa096a7febb0c84e16327056d247e38\": container with ID starting with 0d4a1aa1e9cd5b3562ad3d58571f31758fa096a7febb0c84e16327056d247e38 not found: ID does not exist"
Nov 02 13:06:42.400414 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.400396    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-workload-socket\") pod \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") "
Nov 02 13:06:42.400507 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.400439    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-credential-socket\") pod \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") "
Nov 02 13:06:42.400507 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.400477    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-data\") pod \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") "
Nov 02 13:06:42.400507 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.400496    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-podinfo\") pod \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") "
Nov 02 13:06:42.400684 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.400538    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-kube-api-access-5g5ft\" (UniqueName: \"kubernetes.io/projected/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-kube-api-access-5g5ft\") pod \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") "
Nov 02 13:06:42.400684 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.400603    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-token\" (UniqueName: \"kubernetes.io/projected/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-token\") pod \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") "
Nov 02 13:06:42.400684 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.400654    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-envoy\") pod \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") "
Nov 02 13:06:42.400684 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.400679    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-workload-certs\") pod \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") "
Nov 02 13:06:42.400883 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.400680    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-workload-socket" pod "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" (UID: "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:06:42.400883 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.400709    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istiod-ca-cert\") pod \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\" (UID: \"b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51\") "
Nov 02 13:06:42.400883 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.400764    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-credential-socket" pod "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" (UID: "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:06:42.400883 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.400781    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-data" pod "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" (UID: "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:06:42.401105 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.401053    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-workload-certs" pod "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" (UID: "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:06:42.401160 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.401130    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istiod-ca-cert" pod "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" (UID: "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:42.401250 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.401232    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.401308 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.401255    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.401308 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.401271    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.401308 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.401285    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.401308 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.401301    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.402720 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.402699    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-podinfo" pod "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" (UID: "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:06:42.402814 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.402747    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-kube-api-access-5g5ft" pod "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" (UID: "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51"). InnerVolumeSpecName "kube-api-access-5g5ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:42.402814 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.402750    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-token" pod "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" (UID: "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:42.402814 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.402790    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-envoy" pod "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" (UID: "b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:06:42.502232 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.502197    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-token\" (UniqueName: \"kubernetes.io/projected/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-token\") pod \"f6025f4f-a2b0-4537-8a34-439e39b04658\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") "
Nov 02 13:06:42.502391 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.502237    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-podinfo\") pod \"f6025f4f-a2b0-4537-8a34-439e39b04658\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") "
Nov 02 13:06:42.502391 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.502313    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-credential-socket\") pod \"f6025f4f-a2b0-4537-8a34-439e39b04658\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") "
Nov 02 13:06:42.502391 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.502335    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-envoy\") pod \"f6025f4f-a2b0-4537-8a34-439e39b04658\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") "
Nov 02 13:06:42.502518 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.502409    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-data\") pod \"f6025f4f-a2b0-4537-8a34-439e39b04658\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") "
Nov 02 13:06:42.502518 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.502438    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-workload-socket\") pod \"f6025f4f-a2b0-4537-8a34-439e39b04658\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") "
Nov 02 13:06:42.502518 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.502479    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6025f4f-a2b0-4537-8a34-439e39b04658-kube-api-access-kxzmm\" (UniqueName: \"kubernetes.io/projected/f6025f4f-a2b0-4537-8a34-439e39b04658-kube-api-access-kxzmm\") pod \"f6025f4f-a2b0-4537-8a34-439e39b04658\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") "
Nov 02 13:06:42.502656 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.502532    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6025f4f-a2b0-4537-8a34-439e39b04658-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6025f4f-a2b0-4537-8a34-439e39b04658-istiod-ca-cert\") pod \"f6025f4f-a2b0-4537-8a34-439e39b04658\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") "
Nov 02 13:06:42.502656 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.502558    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-workload-certs\") pod \"f6025f4f-a2b0-4537-8a34-439e39b04658\" (UID: \"f6025f4f-a2b0-4537-8a34-439e39b04658\") "
Nov 02 13:06:42.502656 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.502598    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-credential-socket" pod "f6025f4f-a2b0-4537-8a34-439e39b04658" (UID: "f6025f4f-a2b0-4537-8a34-439e39b04658"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:06:42.502870 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.502846    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-data" pod "f6025f4f-a2b0-4537-8a34-439e39b04658" (UID: "f6025f4f-a2b0-4537-8a34-439e39b04658"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:06:42.502965 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.502919    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.502965 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.502943    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.502965 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.502959    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.503152 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.503000    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.503152 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.503015    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g5ft\" (UniqueName: \"kubernetes.io/projected/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51-kube-api-access-5g5ft\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.503152 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.503050    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-workload-socket" pod "f6025f4f-a2b0-4537-8a34-439e39b04658" (UID: "f6025f4f-a2b0-4537-8a34-439e39b04658"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:06:42.503152 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.503108    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6025f4f-a2b0-4537-8a34-439e39b04658-istiod-ca-cert" pod "f6025f4f-a2b0-4537-8a34-439e39b04658" (UID: "f6025f4f-a2b0-4537-8a34-439e39b04658"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:42.503324 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.503218    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-workload-certs" pod "f6025f4f-a2b0-4537-8a34-439e39b04658" (UID: "f6025f4f-a2b0-4537-8a34-439e39b04658"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:06:42.504511 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.504485    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-envoy" pod "f6025f4f-a2b0-4537-8a34-439e39b04658" (UID: "f6025f4f-a2b0-4537-8a34-439e39b04658"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:06:42.504610 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.504507    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-podinfo" pod "f6025f4f-a2b0-4537-8a34-439e39b04658" (UID: "f6025f4f-a2b0-4537-8a34-439e39b04658"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:06:42.504811 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.504792    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-token" pod "f6025f4f-a2b0-4537-8a34-439e39b04658" (UID: "f6025f4f-a2b0-4537-8a34-439e39b04658"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:42.504949 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.504932    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6025f4f-a2b0-4537-8a34-439e39b04658-kube-api-access-kxzmm" pod "f6025f4f-a2b0-4537-8a34-439e39b04658" (UID: "f6025f4f-a2b0-4537-8a34-439e39b04658"). InnerVolumeSpecName "kube-api-access-kxzmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:42.604111 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.604037    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxzmm\" (UniqueName: \"kubernetes.io/projected/f6025f4f-a2b0-4537-8a34-439e39b04658-kube-api-access-kxzmm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.604111 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.604065    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6025f4f-a2b0-4537-8a34-439e39b04658-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.604111 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.604075    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.604111 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.604084    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.604111 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.604093    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.604111 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.604107    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.604468 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.604118    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.604468 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.604128    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6025f4f-a2b0-4537-8a34-439e39b04658-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:06:42.840075 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.840038    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"]
Nov 02 13:06:42.844754 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:42.844725    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-wktxt"]
Nov 02 13:06:43.382911 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:43.382837    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"
Nov 02 13:06:43.550648 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:43.550603    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"]
Nov 02 13:06:43.555071 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:43.555045    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r6j5r"]
Nov 02 13:06:43.580593 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:43.580568    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" path="/var/lib/kubelet/pods/b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51/volumes"
Nov 02 13:06:43.581044 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:43.581031    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6025f4f-a2b0-4537-8a34-439e39b04658" path="/var/lib/kubelet/pods/f6025f4f-a2b0-4537-8a34-439e39b04658/volumes"
Nov 02 13:06:58.293085 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:06:58.293043    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:06:59.048193 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:59.048155    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"]
Nov 02 13:06:59.098577 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:59.098542    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64aa50e7-fqp22"]
Nov 02 13:06:59.582203 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:06:59.582165    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ba84a2-91ce-4da5-bee6-db5eb46225f2" path="/var/lib/kubelet/pods/91ba84a2-91ce-4da5-bee6-db5eb46225f2/volumes"
Nov 02 13:07:12.047703 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:12.047660    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"]
Nov 02 13:07:12.097985 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:12.097942    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0f2a23-6h8xm"]
Nov 02 13:07:13.586918 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:13.586886    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e37cbdd-16a5-4d30-865c-fee2a8d277a0" path="/var/lib/kubelet/pods/4e37cbdd-16a5-4d30-865c-fee2a8d277a0/volumes"
Nov 02 13:07:14.381072 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:14.381042    4464 scope.go:122] "RemoveContainer" containerID="110e78cc961fb23d6555c21871a1ba1bb31f5f944d8beb303e8bd9d4bbb1199b"
Nov 02 13:07:14.389704 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:14.389681    4464 scope.go:122] "RemoveContainer" containerID="a0a25fae4b32208e0c482fddced2b12d1ccd4ff505bc03b6556a0f10bee4c6e1"
Nov 02 13:07:28.292116 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:07:28.291897    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:07:28.558683 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:28.558596    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-extmod-a-6ac03f_payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz_3dbaad92-d33e-4573-bda3-b5fa373e60e1/payload-processing/0.log"
Nov 02 13:07:28.839641 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:28.839563    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-extmod-b-3a78b4_payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n_69f44410-ee12-4afc-96fb-073cdfd73b73/payload-processing/0.log"
Nov 02 13:07:34.873636 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.873599    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"]
Nov 02 13:07:34.874064 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.874050    4464 cpu_manager.go:414] "containerMap: removing container" podUID="b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" containerName="istio-proxy"
Nov 02 13:07:34.874123 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.874064    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" containerName="istio-proxy"
Nov 02 13:07:34.874123 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.874086    4464 cpu_manager.go:414] "containerMap: removing container" podUID="0ed95fce-2f43-4783-8c0f-20b8702d6004" containerName="payload-processing"
Nov 02 13:07:34.874123 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.874091    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed95fce-2f43-4783-8c0f-20b8702d6004" containerName="payload-processing"
Nov 02 13:07:34.874123 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.874110    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f6025f4f-a2b0-4537-8a34-439e39b04658" containerName="istio-proxy"
Nov 02 13:07:34.874123 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.874115    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6025f4f-a2b0-4537-8a34-439e39b04658" containerName="istio-proxy"
Nov 02 13:07:34.874290 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.874143    4464 cpu_manager.go:414] "containerMap: removing container" podUID="2e052c7c-9a9d-482d-8757-1b160b965466" containerName="istio-proxy"
Nov 02 13:07:34.874290 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.874150    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e052c7c-9a9d-482d-8757-1b160b965466" containerName="istio-proxy"
Nov 02 13:07:34.874290 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.874165    4464 cpu_manager.go:414] "containerMap: removing container" podUID="95108efb-2930-4c24-89f1-b68a3328264b" containerName="payload-processing"
Nov 02 13:07:34.874290 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.874173    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95108efb-2930-4c24-89f1-b68a3328264b" containerName="payload-processing"
Nov 02 13:07:34.874290 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.874283    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6025f4f-a2b0-4537-8a34-439e39b04658" containerName="istio-proxy"
Nov 02 13:07:34.874503 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.874298    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e052c7c-9a9d-482d-8757-1b160b965466" containerName="istio-proxy"
Nov 02 13:07:34.874503 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.874309    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="95108efb-2930-4c24-89f1-b68a3328264b" containerName="payload-processing"
Nov 02 13:07:34.874503 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.874322    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="b595a6ad-3c8e-4a19-b22e-20b5b3fe2b51" containerName="istio-proxy"
Nov 02 13:07:34.874503 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.874336    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed95fce-2f43-4783-8c0f-20b8702d6004" containerName="payload-processing"
Nov 02 13:07:34.877478 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.877459    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:07:34.933474 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.933440    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-4c050dcb5f96\""
Nov 02 13:07:34.985602 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:34.985572    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"]
Nov 02 13:07:35.012663 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:35.012630    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq8l8\" (UniqueName: \"kubernetes.io/projected/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-kube-api-access-rq8l8\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc\" (UID: \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:07:35.012803 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:35.012666    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc\" (UID: \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:07:35.012803 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:35.012695    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc\" (UID: \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:07:35.012879 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:35.012809    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc\" (UID: \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:07:35.113471 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:35.113434    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq8l8\" (UniqueName: \"kubernetes.io/projected/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-kube-api-access-rq8l8\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc\" (UID: \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:07:35.113471 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:35.113475    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc\" (UID: \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:07:35.113682 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:35.113499    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc\" (UID: \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:07:35.113682 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:35.113534    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc\" (UID: \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:07:35.114167 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:35.114140    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc\" (UID: \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:07:35.114300 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:35.114224    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc\" (UID: \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:07:35.116220 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:35.116191    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc\" (UID: \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:07:35.175046 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:35.174959    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq8l8\" (UniqueName: \"kubernetes.io/projected/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-kube-api-access-rq8l8\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc\" (UID: \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:07:35.190801 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:35.190777    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:07:35.367674 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:35.367629    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:07:35.567145 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:35.567106    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"]
Nov 02 13:07:35.569210 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:07:35.569180    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7431df_a6f0_42b3_9fcf_fea3eb07f4dc.slice/crio-e91f7002d1264198a5dddee924734a8129301503e0ab90f1224e4806f36511d2 WatchSource:0}: Error finding container e91f7002d1264198a5dddee924734a8129301503e0ab90f1224e4806f36511d2: Status 404 returned error can't find the container with id e91f7002d1264198a5dddee924734a8129301503e0ab90f1224e4806f36511d2
Nov 02 13:07:35.650998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:35.648403    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc" event={"ID":"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc","Type":"ContainerStarted","Data":"e91f7002d1264198a5dddee924734a8129301503e0ab90f1224e4806f36511d2"}
Nov 02 13:07:36.656648 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:36.656608    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc" event={"ID":"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc","Type":"ContainerStarted","Data":"893a4056bdc1394fb4e4654deb2306c322863aa0ae36201157af29cdad4362e1"}
Nov 02 13:07:36.657088 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:36.656686    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:07:36.776021 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:36.775951    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc" podStartSLOduration=2.775938659 podStartE2EDuration="2.775938659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:07:36.774660514 +0000 UTC m=+4285.338368924" watchObservedRunningTime="2026-11-02 13:07:36.775938659 +0000 UTC m=+4285.339647088"
Nov 02 13:07:37.481709 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:37.481680    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"]
Nov 02 13:07:37.485419 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:37.485402    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:07:37.537834 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:37.537804    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b\" (UID: \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:07:37.537945 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:37.537866    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm5q8\" (UniqueName: \"kubernetes.io/projected/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-kube-api-access-gm5q8\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b\" (UID: \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:07:37.538027 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:37.537935    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b\" (UID: \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:07:37.538027 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:37.537966    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b\" (UID: \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:07:37.540904 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:37.540877    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"agc-cred-2a587a2bd360\""
Nov 02 13:07:37.543279 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:37.543258    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"]
Nov 02 13:07:37.639175 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:37.639142    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b\" (UID: \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:07:37.639175 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:37.639178    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b\" (UID: \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:07:37.639385 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:37.639249    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b\" (UID: \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:07:37.639385 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:37.639310    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm5q8\" (UniqueName: \"kubernetes.io/projected/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-kube-api-access-gm5q8\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b\" (UID: \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:07:37.639893 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:37.639866    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b\" (UID: \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:07:37.640054 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:37.640036    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b\" (UID: \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:07:37.641721 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:37.641695    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b\" (UID: \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:07:37.701763 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:37.701735    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm5q8\" (UniqueName: \"kubernetes.io/projected/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-kube-api-access-gm5q8\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b\" (UID: \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:07:37.798809 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:37.798723    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:07:38.172331 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:38.172298    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"]
Nov 02 13:07:38.173651 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:07:38.173624    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5230cf6f_ed7a_46e9_ae9b_2736a4856ab8.slice/crio-cec710a8a97573dbf8fe4c673bce693757477e7072bf29f44b393c97733c863a WatchSource:0}: Error finding container cec710a8a97573dbf8fe4c673bce693757477e7072bf29f44b393c97733c863a: Status 404 returned error can't find the container with id cec710a8a97573dbf8fe4c673bce693757477e7072bf29f44b393c97733c863a
Nov 02 13:07:38.667961 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:38.667925    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b" event={"ID":"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8","Type":"ContainerStarted","Data":"a9990a3c389d7a567d07ca215c144aefb2d86e460bd3e0fc4011f937079105b4"}
Nov 02 13:07:38.668165 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:38.667998    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b" event={"ID":"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8","Type":"ContainerStarted","Data":"cec710a8a97573dbf8fe4c673bce693757477e7072bf29f44b393c97733c863a"}
Nov 02 13:07:38.668165 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:38.668110    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:07:38.787171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:38.787115    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b" podStartSLOduration=1.787101128 podStartE2EDuration="1.787101128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:07:38.786202397 +0000 UTC m=+4287.349910824" watchObservedRunningTime="2026-11-02 13:07:38.787101128 +0000 UTC m=+4287.350809570"
Nov 02 13:07:47.664721 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:47.664693    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:07:47.836415 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:47.836380    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"]
Nov 02 13:07:49.675638 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:49.675593    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:07:49.847257 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:49.847224    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"]
Nov 02 13:07:52.837460 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:52.837415    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n" podUID="69f44410-ee12-4afc-96fb-073cdfd73b73" containerName="payload-processing" containerID="cri-o://fa8436d5e1f22aedb6ab38e1b09198cba75251677e6dd3e69338c343b0892129" gracePeriod=25
Nov 02 13:07:53.089434 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.089382    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:07:53.190208 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.190173    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f44410-ee12-4afc-96fb-073cdfd73b73-kube-api-access-7fzwm\" (UniqueName: \"kubernetes.io/projected/69f44410-ee12-4afc-96fb-073cdfd73b73-kube-api-access-7fzwm\") pod \"69f44410-ee12-4afc-96fb-073cdfd73b73\" (UID: \"69f44410-ee12-4afc-96fb-073cdfd73b73\") "
Nov 02 13:07:53.190515 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.190296    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f44410-ee12-4afc-96fb-073cdfd73b73-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69f44410-ee12-4afc-96fb-073cdfd73b73-routing-overlay-volume\") pod \"69f44410-ee12-4afc-96fb-073cdfd73b73\" (UID: \"69f44410-ee12-4afc-96fb-073cdfd73b73\") "
Nov 02 13:07:53.190515 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.190339    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f44410-ee12-4afc-96fb-073cdfd73b73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f44410-ee12-4afc-96fb-073cdfd73b73-plugins-config-volume\") pod \"69f44410-ee12-4afc-96fb-073cdfd73b73\" (UID: \"69f44410-ee12-4afc-96fb-073cdfd73b73\") "
Nov 02 13:07:53.190515 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.190394    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f44410-ee12-4afc-96fb-073cdfd73b73-provider-credentials\" (UniqueName: \"kubernetes.io/projected/69f44410-ee12-4afc-96fb-073cdfd73b73-provider-credentials\") pod \"69f44410-ee12-4afc-96fb-073cdfd73b73\" (UID: \"69f44410-ee12-4afc-96fb-073cdfd73b73\") "
Nov 02 13:07:53.190747 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.190684    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f44410-ee12-4afc-96fb-073cdfd73b73-routing-overlay-volume" pod "69f44410-ee12-4afc-96fb-073cdfd73b73" (UID: "69f44410-ee12-4afc-96fb-073cdfd73b73"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:53.190871 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.190854    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69f44410-ee12-4afc-96fb-073cdfd73b73-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:07:53.190963 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.190943    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f44410-ee12-4afc-96fb-073cdfd73b73-plugins-config-volume" pod "69f44410-ee12-4afc-96fb-073cdfd73b73" (UID: "69f44410-ee12-4afc-96fb-073cdfd73b73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:53.192729 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.192707    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f44410-ee12-4afc-96fb-073cdfd73b73-kube-api-access-7fzwm" pod "69f44410-ee12-4afc-96fb-073cdfd73b73" (UID: "69f44410-ee12-4afc-96fb-073cdfd73b73"). InnerVolumeSpecName "kube-api-access-7fzwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:53.192829 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.192737    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f44410-ee12-4afc-96fb-073cdfd73b73-provider-credentials" pod "69f44410-ee12-4afc-96fb-073cdfd73b73" (UID: "69f44410-ee12-4afc-96fb-073cdfd73b73"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:53.291795 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.291760    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fzwm\" (UniqueName: \"kubernetes.io/projected/69f44410-ee12-4afc-96fb-073cdfd73b73-kube-api-access-7fzwm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:07:53.291795 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.291788    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f44410-ee12-4afc-96fb-073cdfd73b73-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:07:53.291795 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.291798    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69f44410-ee12-4afc-96fb-073cdfd73b73-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:07:53.745692 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.745656    4464 generic.go:356] "Generic (PLEG): container finished" podID="69f44410-ee12-4afc-96fb-073cdfd73b73" containerID="fa8436d5e1f22aedb6ab38e1b09198cba75251677e6dd3e69338c343b0892129" exitCode=0
Nov 02 13:07:53.745874 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.745705    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"
Nov 02 13:07:53.745874 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.745734    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n" event={"ID":"69f44410-ee12-4afc-96fb-073cdfd73b73","Type":"ContainerDied","Data":"fa8436d5e1f22aedb6ab38e1b09198cba75251677e6dd3e69338c343b0892129"}
Nov 02 13:07:53.745874 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.745778    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n" event={"ID":"69f44410-ee12-4afc-96fb-073cdfd73b73","Type":"ContainerDied","Data":"5d77843959ae89ef8c250b7bde2a7e4d8590b061376ec72fc05672f89b58599b"}
Nov 02 13:07:53.745874 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.745799    4464 scope.go:122] "RemoveContainer" containerID="fa8436d5e1f22aedb6ab38e1b09198cba75251677e6dd3e69338c343b0892129"
Nov 02 13:07:53.754393 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.754372    4464 scope.go:122] "RemoveContainer" containerID="fa8436d5e1f22aedb6ab38e1b09198cba75251677e6dd3e69338c343b0892129"
Nov 02 13:07:53.754671 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:07:53.754651    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa8436d5e1f22aedb6ab38e1b09198cba75251677e6dd3e69338c343b0892129\": container with ID starting with fa8436d5e1f22aedb6ab38e1b09198cba75251677e6dd3e69338c343b0892129 not found: ID does not exist" containerID="fa8436d5e1f22aedb6ab38e1b09198cba75251677e6dd3e69338c343b0892129"
Nov 02 13:07:53.754761 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.754678    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa8436d5e1f22aedb6ab38e1b09198cba75251677e6dd3e69338c343b0892129"} err="failed to get container status \"fa8436d5e1f22aedb6ab38e1b09198cba75251677e6dd3e69338c343b0892129\": rpc error: code = NotFound desc = could not find container \"fa8436d5e1f22aedb6ab38e1b09198cba75251677e6dd3e69338c343b0892129\": container with ID starting with fa8436d5e1f22aedb6ab38e1b09198cba75251677e6dd3e69338c343b0892129 not found: ID does not exist"
Nov 02 13:07:53.912853 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.912769    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"]
Nov 02 13:07:53.917367 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:53.917340    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589cd7l2n"]
Nov 02 13:07:54.848535 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:54.848498    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz" podUID="3dbaad92-d33e-4573-bda3-b5fa373e60e1" containerName="payload-processing" containerID="cri-o://c7e02363915d6321f0510ce08deb32896aca9553f01575cf6fa8c55d8bd7cdee" gracePeriod=25
Nov 02 13:07:55.086624 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.086597    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:07:55.212464 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.212381    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbaad92-d33e-4573-bda3-b5fa373e60e1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3dbaad92-d33e-4573-bda3-b5fa373e60e1-routing-overlay-volume\") pod \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\" (UID: \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\") "
Nov 02 13:07:55.212632 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.212465    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbaad92-d33e-4573-bda3-b5fa373e60e1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3dbaad92-d33e-4573-bda3-b5fa373e60e1-provider-credentials\") pod \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\" (UID: \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\") "
Nov 02 13:07:55.212632 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.212514    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbaad92-d33e-4573-bda3-b5fa373e60e1-kube-api-access-s62fp\" (UniqueName: \"kubernetes.io/projected/3dbaad92-d33e-4573-bda3-b5fa373e60e1-kube-api-access-s62fp\") pod \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\" (UID: \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\") "
Nov 02 13:07:55.212632 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.212543    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbaad92-d33e-4573-bda3-b5fa373e60e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dbaad92-d33e-4573-bda3-b5fa373e60e1-plugins-config-volume\") pod \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\" (UID: \"3dbaad92-d33e-4573-bda3-b5fa373e60e1\") "
Nov 02 13:07:55.212802 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.212731    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbaad92-d33e-4573-bda3-b5fa373e60e1-routing-overlay-volume" pod "3dbaad92-d33e-4573-bda3-b5fa373e60e1" (UID: "3dbaad92-d33e-4573-bda3-b5fa373e60e1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:55.212941 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.212922    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3dbaad92-d33e-4573-bda3-b5fa373e60e1-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:07:55.213118 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.213075    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbaad92-d33e-4573-bda3-b5fa373e60e1-plugins-config-volume" pod "3dbaad92-d33e-4573-bda3-b5fa373e60e1" (UID: "3dbaad92-d33e-4573-bda3-b5fa373e60e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:55.214845 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.214809    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbaad92-d33e-4573-bda3-b5fa373e60e1-provider-credentials" pod "3dbaad92-d33e-4573-bda3-b5fa373e60e1" (UID: "3dbaad92-d33e-4573-bda3-b5fa373e60e1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:55.214955 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.214836    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbaad92-d33e-4573-bda3-b5fa373e60e1-kube-api-access-s62fp" pod "3dbaad92-d33e-4573-bda3-b5fa373e60e1" (UID: "3dbaad92-d33e-4573-bda3-b5fa373e60e1"). InnerVolumeSpecName "kube-api-access-s62fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:55.313489 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.313459    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3dbaad92-d33e-4573-bda3-b5fa373e60e1-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:07:55.313489 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.313489    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s62fp\" (UniqueName: \"kubernetes.io/projected/3dbaad92-d33e-4573-bda3-b5fa373e60e1-kube-api-access-s62fp\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:07:55.313658 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.313505    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dbaad92-d33e-4573-bda3-b5fa373e60e1-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:07:55.580719 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.580684    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f44410-ee12-4afc-96fb-073cdfd73b73" path="/var/lib/kubelet/pods/69f44410-ee12-4afc-96fb-073cdfd73b73/volumes"
Nov 02 13:07:55.698151 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.698123    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"]
Nov 02 13:07:55.698576 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.698564    4464 cpu_manager.go:414] "containerMap: removing container" podUID="69f44410-ee12-4afc-96fb-073cdfd73b73" containerName="payload-processing"
Nov 02 13:07:55.698621 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.698577    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f44410-ee12-4afc-96fb-073cdfd73b73" containerName="payload-processing"
Nov 02 13:07:55.698621 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.698598    4464 cpu_manager.go:414] "containerMap: removing container" podUID="3dbaad92-d33e-4573-bda3-b5fa373e60e1" containerName="payload-processing"
Nov 02 13:07:55.698621 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.698603    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbaad92-d33e-4573-bda3-b5fa373e60e1" containerName="payload-processing"
Nov 02 13:07:55.698726 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.698670    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbaad92-d33e-4573-bda3-b5fa373e60e1" containerName="payload-processing"
Nov 02 13:07:55.698726 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.698686    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f44410-ee12-4afc-96fb-073cdfd73b73" containerName="payload-processing"
Nov 02 13:07:55.701860 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.701843    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:07:55.757805 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.757779    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-59685836c2a5\""
Nov 02 13:07:55.760448 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.760426    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"]
Nov 02 13:07:55.761086 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.761063    4464 generic.go:356] "Generic (PLEG): container finished" podID="3dbaad92-d33e-4573-bda3-b5fa373e60e1" containerID="c7e02363915d6321f0510ce08deb32896aca9553f01575cf6fa8c55d8bd7cdee" exitCode=0
Nov 02 13:07:55.761184 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.761106    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"
Nov 02 13:07:55.761184 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.761130    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz" event={"ID":"3dbaad92-d33e-4573-bda3-b5fa373e60e1","Type":"ContainerDied","Data":"c7e02363915d6321f0510ce08deb32896aca9553f01575cf6fa8c55d8bd7cdee"}
Nov 02 13:07:55.761184 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.761166    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz" event={"ID":"3dbaad92-d33e-4573-bda3-b5fa373e60e1","Type":"ContainerDied","Data":"1e1dc87611ff982814a634846d49821a8a3ccacb44a9e7b234297dd8a37fe784"}
Nov 02 13:07:55.761302 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.761192    4464 scope.go:122] "RemoveContainer" containerID="c7e02363915d6321f0510ce08deb32896aca9553f01575cf6fa8c55d8bd7cdee"
Nov 02 13:07:55.777135 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.777116    4464 scope.go:122] "RemoveContainer" containerID="c7e02363915d6321f0510ce08deb32896aca9553f01575cf6fa8c55d8bd7cdee"
Nov 02 13:07:55.777457 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:07:55.777430    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e02363915d6321f0510ce08deb32896aca9553f01575cf6fa8c55d8bd7cdee\": container with ID starting with c7e02363915d6321f0510ce08deb32896aca9553f01575cf6fa8c55d8bd7cdee not found: ID does not exist" containerID="c7e02363915d6321f0510ce08deb32896aca9553f01575cf6fa8c55d8bd7cdee"
Nov 02 13:07:55.777569 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.777461    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e02363915d6321f0510ce08deb32896aca9553f01575cf6fa8c55d8bd7cdee"} err="failed to get container status \"c7e02363915d6321f0510ce08deb32896aca9553f01575cf6fa8c55d8bd7cdee\": rpc error: code = NotFound desc = could not find container \"c7e02363915d6321f0510ce08deb32896aca9553f01575cf6fa8c55d8bd7cdee\": container with ID starting with c7e02363915d6321f0510ce08deb32896aca9553f01575cf6fa8c55d8bd7cdee not found: ID does not exist"
Nov 02 13:07:55.818375 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.818346    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp7xc\" (UniqueName: \"kubernetes.io/projected/0c8eebe5-2c24-49c9-abd3-2890b0969071-kube-api-access-fp7xc\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx\" (UID: \"0c8eebe5-2c24-49c9-abd3-2890b0969071\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:07:55.818477 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.818390    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c8eebe5-2c24-49c9-abd3-2890b0969071-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx\" (UID: \"0c8eebe5-2c24-49c9-abd3-2890b0969071\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:07:55.818477 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.818418    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c8eebe5-2c24-49c9-abd3-2890b0969071-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx\" (UID: \"0c8eebe5-2c24-49c9-abd3-2890b0969071\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:07:55.818553 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.818520    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c8eebe5-2c24-49c9-abd3-2890b0969071-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx\" (UID: \"0c8eebe5-2c24-49c9-abd3-2890b0969071\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:07:55.919145 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.919047    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c8eebe5-2c24-49c9-abd3-2890b0969071-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx\" (UID: \"0c8eebe5-2c24-49c9-abd3-2890b0969071\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:07:55.919296 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.919163    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp7xc\" (UniqueName: \"kubernetes.io/projected/0c8eebe5-2c24-49c9-abd3-2890b0969071-kube-api-access-fp7xc\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx\" (UID: \"0c8eebe5-2c24-49c9-abd3-2890b0969071\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:07:55.919296 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.919199    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c8eebe5-2c24-49c9-abd3-2890b0969071-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx\" (UID: \"0c8eebe5-2c24-49c9-abd3-2890b0969071\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:07:55.919296 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.919234    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c8eebe5-2c24-49c9-abd3-2890b0969071-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx\" (UID: \"0c8eebe5-2c24-49c9-abd3-2890b0969071\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:07:55.919952 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.919926    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c8eebe5-2c24-49c9-abd3-2890b0969071-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx\" (UID: \"0c8eebe5-2c24-49c9-abd3-2890b0969071\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:07:55.920167 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.920075    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c8eebe5-2c24-49c9-abd3-2890b0969071-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx\" (UID: \"0c8eebe5-2c24-49c9-abd3-2890b0969071\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:07:55.921917 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.921897    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c8eebe5-2c24-49c9-abd3-2890b0969071-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx\" (UID: \"0c8eebe5-2c24-49c9-abd3-2890b0969071\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:07:55.935436 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.935415    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"]
Nov 02 13:07:55.939783 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.939760    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78lqxmz"]
Nov 02 13:07:55.981320 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:55.981295    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp7xc\" (UniqueName: \"kubernetes.io/projected/0c8eebe5-2c24-49c9-abd3-2890b0969071-kube-api-access-fp7xc\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx\" (UID: \"0c8eebe5-2c24-49c9-abd3-2890b0969071\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:07:56.014247 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:56.014224    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:07:56.387814 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:56.387785    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"]
Nov 02 13:07:56.389710 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:07:56.389683    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c8eebe5_2c24_49c9_abd3_2890b0969071.slice/crio-6571b3723b7af37e6a8e870027bac23f9412bfb75282c4a0ddad528ceb6dd74c WatchSource:0}: Error finding container 6571b3723b7af37e6a8e870027bac23f9412bfb75282c4a0ddad528ceb6dd74c: Status 404 returned error can't find the container with id 6571b3723b7af37e6a8e870027bac23f9412bfb75282c4a0ddad528ceb6dd74c
Nov 02 13:07:56.768114 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:56.768071    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx" event={"ID":"0c8eebe5-2c24-49c9-abd3-2890b0969071","Type":"ContainerStarted","Data":"4ba05b6da2b4cce4eb7333380e8f1d849bce6d26671cfe5977e600595c6273a9"}
Nov 02 13:07:56.768314 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:56.768128    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx" event={"ID":"0c8eebe5-2c24-49c9-abd3-2890b0969071","Type":"ContainerStarted","Data":"6571b3723b7af37e6a8e870027bac23f9412bfb75282c4a0ddad528ceb6dd74c"}
Nov 02 13:07:56.768314 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:56.768155    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:07:56.824601 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:56.816622    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:07:56.894090 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:56.894024    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx" podStartSLOduration=1.894003583 podStartE2EDuration="1.894003583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:07:56.891191323 +0000 UTC m=+4305.454899745" watchObservedRunningTime="2026-11-02 13:07:56.894003583 +0000 UTC m=+4305.457711994"
Nov 02 13:07:57.581358 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:57.581324    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbaad92-d33e-4573-bda3-b5fa373e60e1" path="/var/lib/kubelet/pods/3dbaad92-d33e-4573-bda3-b5fa373e60e1/volumes"
Nov 02 13:07:58.292851 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:07:58.292815    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:07:58.298042 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:58.297954    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"]
Nov 02 13:07:58.322570 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:58.303190    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:07:58.358013 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:58.357966    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6ac03f\"/\"agc-cred-32cb2b2dfd51\""
Nov 02 13:07:58.411238 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:58.411207    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"]
Nov 02 13:07:58.448522 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:58.448487    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4696e4-c36d-4e16-bb64-cb1697e320ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w\" (UID: \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:07:58.448686 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:58.448533    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w\" (UID: \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:07:58.448686 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:58.448590    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b4696e4-c36d-4e16-bb64-cb1697e320ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w\" (UID: \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:07:58.448763 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:58.448716    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2hbx\" (UniqueName: \"kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-kube-api-access-x2hbx\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w\" (UID: \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:07:58.549674 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:58.549599    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b4696e4-c36d-4e16-bb64-cb1697e320ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w\" (UID: \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:07:58.549844 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:58.549688    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2hbx\" (UniqueName: \"kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-kube-api-access-x2hbx\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w\" (UID: \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:07:58.549844 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:58.549721    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4696e4-c36d-4e16-bb64-cb1697e320ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w\" (UID: \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:07:58.549844 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:58.549743    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w\" (UID: \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:07:58.550309 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:58.550284    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b4696e4-c36d-4e16-bb64-cb1697e320ce-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w\" (UID: \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:07:58.550528 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:58.550508    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4696e4-c36d-4e16-bb64-cb1697e320ce-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w\" (UID: \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:07:58.552300 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:58.552274    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w\" (UID: \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:07:58.622454 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:58.622418    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2hbx\" (UniqueName: \"kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-kube-api-access-x2hbx\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w\" (UID: \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:07:58.917851 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:58.917768    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:07:59.291183 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:59.291154    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"]
Nov 02 13:07:59.292579 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:07:59.292550    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4696e4_c36d_4e16_bb64_cb1697e320ce.slice/crio-992cdc61084516b45a06b69be3065694057b8baf9e8708d3aefa549f983c4e75 WatchSource:0}: Error finding container 992cdc61084516b45a06b69be3065694057b8baf9e8708d3aefa549f983c4e75: Status 404 returned error can't find the container with id 992cdc61084516b45a06b69be3065694057b8baf9e8708d3aefa549f983c4e75
Nov 02 13:07:59.787308 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:59.787273    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w" event={"ID":"7b4696e4-c36d-4e16-bb64-cb1697e320ce","Type":"ContainerStarted","Data":"000b755a77737abf603701afea6768f6b80c36c01ea607333d61a81dc50f8198"}
Nov 02 13:07:59.787733 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:59.787326    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w" event={"ID":"7b4696e4-c36d-4e16-bb64-cb1697e320ce","Type":"ContainerStarted","Data":"992cdc61084516b45a06b69be3065694057b8baf9e8708d3aefa549f983c4e75"}
Nov 02 13:07:59.787733 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:59.787349    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:07:59.908388 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:07:59.908336    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w" podStartSLOduration=1.908323596 podStartE2EDuration="1.908323596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:07:59.906373694 +0000 UTC m=+4308.470082139" watchObservedRunningTime="2026-11-02 13:07:59.908323596 +0000 UTC m=+4308.472032040"
Nov 02 13:08:07.778248 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:07.778217    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:08:07.950474 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:07.950442    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"]
Nov 02 13:08:10.795923 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:10.795893    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:08:10.968422 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:10.968382    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"]
Nov 02 13:08:12.951480 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:12.951429    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc" podUID="9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc" containerName="payload-processing" containerID="cri-o://893a4056bdc1394fb4e4654deb2306c322863aa0ae36201157af29cdad4362e1" gracePeriod=25
Nov 02 13:08:13.206920 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.206856    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:08:13.396731 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.396692    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-provider-credentials\") pod \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\" (UID: \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\") "
Nov 02 13:08:13.396934 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.396810    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-kube-api-access-rq8l8\" (UniqueName: \"kubernetes.io/projected/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-kube-api-access-rq8l8\") pod \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\" (UID: \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\") "
Nov 02 13:08:13.396934 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.396866    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-plugins-config-volume\") pod \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\" (UID: \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\") "
Nov 02 13:08:13.396934 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.396893    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-routing-overlay-volume\") pod \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\" (UID: \"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc\") "
Nov 02 13:08:13.397339 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.397310    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-plugins-config-volume" pod "9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc" (UID: "9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:13.397470 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.397316    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-routing-overlay-volume" pod "9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc" (UID: "9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:13.398817 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.398793    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-provider-credentials" pod "9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc" (UID: "9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:13.398906 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.398835    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-kube-api-access-rq8l8" pod "9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc" (UID: "9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc"). InnerVolumeSpecName "kube-api-access-rq8l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:13.498182 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.498145    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:08:13.498182 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.498179    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:08:13.498403 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.498192    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq8l8\" (UniqueName: \"kubernetes.io/projected/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-kube-api-access-rq8l8\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:08:13.498403 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.498204    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:08:13.862141 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.862060    4464 generic.go:356] "Generic (PLEG): container finished" podID="9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc" containerID="893a4056bdc1394fb4e4654deb2306c322863aa0ae36201157af29cdad4362e1" exitCode=0
Nov 02 13:08:13.862141 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.862112    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"
Nov 02 13:08:13.862341 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.862142    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc" event={"ID":"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc","Type":"ContainerDied","Data":"893a4056bdc1394fb4e4654deb2306c322863aa0ae36201157af29cdad4362e1"}
Nov 02 13:08:13.862341 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.862197    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc" event={"ID":"9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc","Type":"ContainerDied","Data":"e91f7002d1264198a5dddee924734a8129301503e0ab90f1224e4806f36511d2"}
Nov 02 13:08:13.862341 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.862220    4464 scope.go:122] "RemoveContainer" containerID="893a4056bdc1394fb4e4654deb2306c322863aa0ae36201157af29cdad4362e1"
Nov 02 13:08:13.871046 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.871023    4464 scope.go:122] "RemoveContainer" containerID="893a4056bdc1394fb4e4654deb2306c322863aa0ae36201157af29cdad4362e1"
Nov 02 13:08:13.871345 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:08:13.871324    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"893a4056bdc1394fb4e4654deb2306c322863aa0ae36201157af29cdad4362e1\": container with ID starting with 893a4056bdc1394fb4e4654deb2306c322863aa0ae36201157af29cdad4362e1 not found: ID does not exist" containerID="893a4056bdc1394fb4e4654deb2306c322863aa0ae36201157af29cdad4362e1"
Nov 02 13:08:13.871409 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:13.871356    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893a4056bdc1394fb4e4654deb2306c322863aa0ae36201157af29cdad4362e1"} err="failed to get container status \"893a4056bdc1394fb4e4654deb2306c322863aa0ae36201157af29cdad4362e1\": rpc error: code = NotFound desc = could not find container \"893a4056bdc1394fb4e4654deb2306c322863aa0ae36201157af29cdad4362e1\": container with ID starting with 893a4056bdc1394fb4e4654deb2306c322863aa0ae36201157af29cdad4362e1 not found: ID does not exist"
Nov 02 13:08:14.030354 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:14.030324    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"]
Nov 02 13:08:14.081270 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:14.081238    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f842crcc"]
Nov 02 13:08:14.424488 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:14.424453    4464 scope.go:122] "RemoveContainer" containerID="1e27b8d6f7dd6387329d2b999881eed3d231c230e5da41181d321be60910736f"
Nov 02 13:08:14.433510 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:14.433480    4464 scope.go:122] "RemoveContainer" containerID="b86031be795ba03a15e968d027404f21bb4e072b9f54601c194c49ed010081bb"
Nov 02 13:08:15.581563 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:15.581523    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc" path="/var/lib/kubelet/pods/9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc/volumes"
Nov 02 13:08:15.968913 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:15.968876    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b" podUID="5230cf6f-ed7a-46e9-ae9b-2736a4856ab8" containerName="payload-processing" containerID="cri-o://a9990a3c389d7a567d07ca215c144aefb2d86e460bd3e0fc4011f937079105b4" gracePeriod=25
Nov 02 13:08:16.215316 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.215294    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:08:16.224968 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.224914    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-routing-overlay-volume\") pod \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\" (UID: \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\") "
Nov 02 13:08:16.224968 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.224955    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-provider-credentials\") pod \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\" (UID: \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\") "
Nov 02 13:08:16.225171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.225016    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-plugins-config-volume\") pod \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\" (UID: \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\") "
Nov 02 13:08:16.225171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.225119    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-kube-api-access-gm5q8\" (UniqueName: \"kubernetes.io/projected/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-kube-api-access-gm5q8\") pod \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\" (UID: \"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8\") "
Nov 02 13:08:16.225421 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.225389    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-routing-overlay-volume" pod "5230cf6f-ed7a-46e9-ae9b-2736a4856ab8" (UID: "5230cf6f-ed7a-46e9-ae9b-2736a4856ab8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:16.225550 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.225527    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-plugins-config-volume" pod "5230cf6f-ed7a-46e9-ae9b-2736a4856ab8" (UID: "5230cf6f-ed7a-46e9-ae9b-2736a4856ab8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:16.227225 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.227200    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-kube-api-access-gm5q8" pod "5230cf6f-ed7a-46e9-ae9b-2736a4856ab8" (UID: "5230cf6f-ed7a-46e9-ae9b-2736a4856ab8"). InnerVolumeSpecName "kube-api-access-gm5q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:16.227325 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.227307    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-provider-credentials" pod "5230cf6f-ed7a-46e9-ae9b-2736a4856ab8" (UID: "5230cf6f-ed7a-46e9-ae9b-2736a4856ab8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:16.326252 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.326216    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm5q8\" (UniqueName: \"kubernetes.io/projected/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-kube-api-access-gm5q8\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:08:16.326252 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.326246    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:08:16.326459 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.326260    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:08:16.326459 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.326274    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:08:16.878257 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.878222    4464 generic.go:356] "Generic (PLEG): container finished" podID="5230cf6f-ed7a-46e9-ae9b-2736a4856ab8" containerID="a9990a3c389d7a567d07ca215c144aefb2d86e460bd3e0fc4011f937079105b4" exitCode=0
Nov 02 13:08:16.878723 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.878271    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"
Nov 02 13:08:16.878723 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.878288    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b" event={"ID":"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8","Type":"ContainerDied","Data":"a9990a3c389d7a567d07ca215c144aefb2d86e460bd3e0fc4011f937079105b4"}
Nov 02 13:08:16.878723 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.878342    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b" event={"ID":"5230cf6f-ed7a-46e9-ae9b-2736a4856ab8","Type":"ContainerDied","Data":"cec710a8a97573dbf8fe4c673bce693757477e7072bf29f44b393c97733c863a"}
Nov 02 13:08:16.878723 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.878374    4464 scope.go:122] "RemoveContainer" containerID="a9990a3c389d7a567d07ca215c144aefb2d86e460bd3e0fc4011f937079105b4"
Nov 02 13:08:16.887730 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.887713    4464 scope.go:122] "RemoveContainer" containerID="a9990a3c389d7a567d07ca215c144aefb2d86e460bd3e0fc4011f937079105b4"
Nov 02 13:08:16.888023 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:08:16.887964    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9990a3c389d7a567d07ca215c144aefb2d86e460bd3e0fc4011f937079105b4\": container with ID starting with a9990a3c389d7a567d07ca215c144aefb2d86e460bd3e0fc4011f937079105b4 not found: ID does not exist" containerID="a9990a3c389d7a567d07ca215c144aefb2d86e460bd3e0fc4011f937079105b4"
Nov 02 13:08:16.888023 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:16.888009    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9990a3c389d7a567d07ca215c144aefb2d86e460bd3e0fc4011f937079105b4"} err="failed to get container status \"a9990a3c389d7a567d07ca215c144aefb2d86e460bd3e0fc4011f937079105b4\": rpc error: code = NotFound desc = could not find container \"a9990a3c389d7a567d07ca215c144aefb2d86e460bd3e0fc4011f937079105b4\": container with ID starting with a9990a3c389d7a567d07ca215c144aefb2d86e460bd3e0fc4011f937079105b4 not found: ID does not exist"
Nov 02 13:08:17.048642 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:17.048609    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"]
Nov 02 13:08:17.099468 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:17.099429    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fddfg9b"]
Nov 02 13:08:17.581278 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:17.581248    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5230cf6f-ed7a-46e9-ae9b-2736a4856ab8" path="/var/lib/kubelet/pods/5230cf6f-ed7a-46e9-ae9b-2736a4856ab8/volumes"
Nov 02 13:08:19.812610 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:19.812566    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:08:28.291729 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:08:28.291694    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:08:31.863645 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:08:31.863606    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:08:58.292132 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:08:58.292047    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:09:28.291655 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:09:28.291618    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:09:58.291768 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:09:58.291733    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:10:28.291590 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:10:28.291503    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:10:58.291283 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:10:58.291237    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:11:12.163296 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:12.163268    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:11:12.181211 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:12.181187    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:11:12.208176 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:12.208153    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:11:12.226024 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:12.226000    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:11:28.291328 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:11:28.291289    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:11:40.355696 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:40.355667    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-extmod-a-6ac03f_payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w_7b4696e4-c36d-4e16-bb64-cb1697e320ce/payload-processing/0.log"
Nov 02 13:11:40.620886 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:40.620804    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-extmod-b-3a78b4_payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx_0c8eebe5-2c24-49c9-abd3-2890b0969071/payload-processing/0.log"
Nov 02 13:11:41.006119 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.006089    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"]
Nov 02 13:11:41.006527 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.006514    4464 cpu_manager.go:414] "containerMap: removing container" podUID="5230cf6f-ed7a-46e9-ae9b-2736a4856ab8" containerName="payload-processing"
Nov 02 13:11:41.006571 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.006528    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5230cf6f-ed7a-46e9-ae9b-2736a4856ab8" containerName="payload-processing"
Nov 02 13:11:41.006571 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.006555    4464 cpu_manager.go:414] "containerMap: removing container" podUID="9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc" containerName="payload-processing"
Nov 02 13:11:41.006571 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.006560    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc" containerName="payload-processing"
Nov 02 13:11:41.006681 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.006645    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="5230cf6f-ed7a-46e9-ae9b-2736a4856ab8" containerName="payload-processing"
Nov 02 13:11:41.006681 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.006657    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f7431df-a6f0-42b3-9fcf-fea3eb07f4dc" containerName="payload-processing"
Nov 02 13:11:41.009783 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.009766    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"
Nov 02 13:11:41.063284 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.063251    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:11:41.113870 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.113834    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:11:41.113870 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.113835    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:11:41.114105 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.113943    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 13:11:41.115968 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.115949    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"]
Nov 02 13:11:41.142395 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.142370    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/407a7564-7d06-4f2a-a433-9e28c23e8550-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd03fbd1-c6kcd\" (UID: \"407a7564-7d06-4f2a-a433-9e28c23e8550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"
Nov 02 13:11:41.142508 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.142400    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lllz6\" (UniqueName: \"kubernetes.io/projected/407a7564-7d06-4f2a-a433-9e28c23e8550-kube-api-access-lllz6\") pod \"maas-api-revoke-sub-dd03fbd1-c6kcd\" (UID: \"407a7564-7d06-4f2a-a433-9e28c23e8550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"
Nov 02 13:11:41.142589 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.142566    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/407a7564-7d06-4f2a-a433-9e28c23e8550-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd03fbd1-c6kcd\" (UID: \"407a7564-7d06-4f2a-a433-9e28c23e8550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"
Nov 02 13:11:41.243766 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.243743    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/407a7564-7d06-4f2a-a433-9e28c23e8550-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd03fbd1-c6kcd\" (UID: \"407a7564-7d06-4f2a-a433-9e28c23e8550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"
Nov 02 13:11:41.243905 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.243793    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/407a7564-7d06-4f2a-a433-9e28c23e8550-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd03fbd1-c6kcd\" (UID: \"407a7564-7d06-4f2a-a433-9e28c23e8550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"
Nov 02 13:11:41.243905 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.243816    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lllz6\" (UniqueName: \"kubernetes.io/projected/407a7564-7d06-4f2a-a433-9e28c23e8550-kube-api-access-lllz6\") pod \"maas-api-revoke-sub-dd03fbd1-c6kcd\" (UID: \"407a7564-7d06-4f2a-a433-9e28c23e8550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"
Nov 02 13:11:41.244501 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.244476    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/407a7564-7d06-4f2a-a433-9e28c23e8550-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd03fbd1-c6kcd\" (UID: \"407a7564-7d06-4f2a-a433-9e28c23e8550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"
Nov 02 13:11:41.246205 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.246181    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/407a7564-7d06-4f2a-a433-9e28c23e8550-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd03fbd1-c6kcd\" (UID: \"407a7564-7d06-4f2a-a433-9e28c23e8550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"
Nov 02 13:11:41.305690 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.305630    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lllz6\" (UniqueName: \"kubernetes.io/projected/407a7564-7d06-4f2a-a433-9e28c23e8550-kube-api-access-lllz6\") pod \"maas-api-revoke-sub-dd03fbd1-c6kcd\" (UID: \"407a7564-7d06-4f2a-a433-9e28c23e8550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"
Nov 02 13:11:41.322380 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.322363    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"
Nov 02 13:11:41.698510 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.698479    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"]
Nov 02 13:11:41.700061 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:11:41.700025    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407a7564_7d06_4f2a_a433_9e28c23e8550.slice/crio-11a2d00067545e66f253f754bf1cf19999f7a3230936f08a6e56328a6bc021c9 WatchSource:0}: Error finding container 11a2d00067545e66f253f754bf1cf19999f7a3230936f08a6e56328a6bc021c9: Status 404 returned error can't find the container with id 11a2d00067545e66f253f754bf1cf19999f7a3230936f08a6e56328a6bc021c9
Nov 02 13:11:41.896719 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:41.896687    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd" event={"ID":"407a7564-7d06-4f2a-a433-9e28c23e8550","Type":"ContainerStarted","Data":"11a2d00067545e66f253f754bf1cf19999f7a3230936f08a6e56328a6bc021c9"}
Nov 02 13:11:42.903789 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:42.903706    4464 generic.go:356] "Generic (PLEG): container finished" podID="407a7564-7d06-4f2a-a433-9e28c23e8550" containerID="7b4919aa105f576ac6ae0169c510c50c09a9ff968645d062ad57c402ac267948" exitCode=0
Nov 02 13:11:42.904201 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:42.903788    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd" event={"ID":"407a7564-7d06-4f2a-a433-9e28c23e8550","Type":"ContainerDied","Data":"7b4919aa105f576ac6ae0169c510c50c09a9ff968645d062ad57c402ac267948"}
Nov 02 13:11:44.043539 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:44.043513    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"
Nov 02 13:11:44.172690 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:44.172615    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/407a7564-7d06-4f2a-a433-9e28c23e8550-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/407a7564-7d06-4f2a-a433-9e28c23e8550-maas-api-xks-ca\") pod \"407a7564-7d06-4f2a-a433-9e28c23e8550\" (UID: \"407a7564-7d06-4f2a-a433-9e28c23e8550\") "
Nov 02 13:11:44.172690 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:44.172681    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407a7564-7d06-4f2a-a433-9e28c23e8550-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/407a7564-7d06-4f2a-a433-9e28c23e8550-maas-api-openshift-ca\") pod \"407a7564-7d06-4f2a-a433-9e28c23e8550\" (UID: \"407a7564-7d06-4f2a-a433-9e28c23e8550\") "
Nov 02 13:11:44.172937 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:44.172728    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407a7564-7d06-4f2a-a433-9e28c23e8550-kube-api-access-lllz6\" (UniqueName: \"kubernetes.io/projected/407a7564-7d06-4f2a-a433-9e28c23e8550-kube-api-access-lllz6\") pod \"407a7564-7d06-4f2a-a433-9e28c23e8550\" (UID: \"407a7564-7d06-4f2a-a433-9e28c23e8550\") "
Nov 02 13:11:44.173130 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:44.173095    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407a7564-7d06-4f2a-a433-9e28c23e8550-maas-api-openshift-ca" pod "407a7564-7d06-4f2a-a433-9e28c23e8550" (UID: "407a7564-7d06-4f2a-a433-9e28c23e8550"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:11:44.174806 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:44.174779    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/407a7564-7d06-4f2a-a433-9e28c23e8550-maas-api-xks-ca" pod "407a7564-7d06-4f2a-a433-9e28c23e8550" (UID: "407a7564-7d06-4f2a-a433-9e28c23e8550"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:11:44.174947 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:44.174917    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407a7564-7d06-4f2a-a433-9e28c23e8550-kube-api-access-lllz6" pod "407a7564-7d06-4f2a-a433-9e28c23e8550" (UID: "407a7564-7d06-4f2a-a433-9e28c23e8550"). InnerVolumeSpecName "kube-api-access-lllz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:11:44.274343 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:44.274311    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/407a7564-7d06-4f2a-a433-9e28c23e8550-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:11:44.274343 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:44.274338    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/407a7564-7d06-4f2a-a433-9e28c23e8550-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:11:44.274528 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:44.274350    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lllz6\" (UniqueName: \"kubernetes.io/projected/407a7564-7d06-4f2a-a433-9e28c23e8550-kube-api-access-lllz6\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:11:44.914400 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:44.914367    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd" event={"ID":"407a7564-7d06-4f2a-a433-9e28c23e8550","Type":"ContainerDied","Data":"11a2d00067545e66f253f754bf1cf19999f7a3230936f08a6e56328a6bc021c9"}
Nov 02 13:11:44.914400 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:44.914384    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"
Nov 02 13:11:44.914629 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:44.914414    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11a2d00067545e66f253f754bf1cf19999f7a3230936f08a6e56328a6bc021c9"
Nov 02 13:11:52.676218 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.676174    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"]
Nov 02 13:11:52.676835 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.676813    4464 cpu_manager.go:414] "containerMap: removing container" podUID="407a7564-7d06-4f2a-a433-9e28c23e8550" containerName="revoke-keys"
Nov 02 13:11:52.676935 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.676837    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407a7564-7d06-4f2a-a433-9e28c23e8550" containerName="revoke-keys"
Nov 02 13:11:52.677033 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.676993    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="407a7564-7d06-4f2a-a433-9e28c23e8550" containerName="revoke-keys"
Nov 02 13:11:52.680305 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.680283    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"
Nov 02 13:11:52.738400 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.738367    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"]
Nov 02 13:11:52.751015 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.750960    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6x7l\" (UniqueName: \"kubernetes.io/projected/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-kube-api-access-z6x7l\") pod \"maas-api-revoke-sub-d41b2555-cztmt\" (UID: \"64e1a02f-bf60-4321-9bd4-1313a8c4fd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"
Nov 02 13:11:52.751151 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.751097    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d41b2555-cztmt\" (UID: \"64e1a02f-bf60-4321-9bd4-1313a8c4fd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"
Nov 02 13:11:52.751202 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.751157    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d41b2555-cztmt\" (UID: \"64e1a02f-bf60-4321-9bd4-1313a8c4fd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"
Nov 02 13:11:52.786656 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.786630    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:11:52.786777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.786667    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:11:52.786777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.786712    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:11:52.786851 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.786783    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 13:11:52.852428 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.852399    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d41b2555-cztmt\" (UID: \"64e1a02f-bf60-4321-9bd4-1313a8c4fd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"
Nov 02 13:11:52.852545 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.852457    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d41b2555-cztmt\" (UID: \"64e1a02f-bf60-4321-9bd4-1313a8c4fd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"
Nov 02 13:11:52.852545 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.852536    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6x7l\" (UniqueName: \"kubernetes.io/projected/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-kube-api-access-z6x7l\") pod \"maas-api-revoke-sub-d41b2555-cztmt\" (UID: \"64e1a02f-bf60-4321-9bd4-1313a8c4fd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"
Nov 02 13:11:52.853196 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.853177    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d41b2555-cztmt\" (UID: \"64e1a02f-bf60-4321-9bd4-1313a8c4fd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"
Nov 02 13:11:52.854679 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.854659    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d41b2555-cztmt\" (UID: \"64e1a02f-bf60-4321-9bd4-1313a8c4fd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"
Nov 02 13:11:52.914361 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.914339    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6x7l\" (UniqueName: \"kubernetes.io/projected/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-kube-api-access-z6x7l\") pod \"maas-api-revoke-sub-d41b2555-cztmt\" (UID: \"64e1a02f-bf60-4321-9bd4-1313a8c4fd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"
Nov 02 13:11:52.993457 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:52.993428    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"
Nov 02 13:11:53.379359 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:53.379334    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"]
Nov 02 13:11:53.380592 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:11:53.380569    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e1a02f_bf60_4321_9bd4_1313a8c4fd55.slice/crio-8ffc6d4ccac850fb0ff10779efcf708e99a4f9e7de2235e443869dd06ca3883e WatchSource:0}: Error finding container 8ffc6d4ccac850fb0ff10779efcf708e99a4f9e7de2235e443869dd06ca3883e: Status 404 returned error can't find the container with id 8ffc6d4ccac850fb0ff10779efcf708e99a4f9e7de2235e443869dd06ca3883e
Nov 02 13:11:53.962625 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:53.962572    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt" event={"ID":"64e1a02f-bf60-4321-9bd4-1313a8c4fd55","Type":"ContainerStarted","Data":"14b159824bc37024ea1d387506b9a5e98f169b4d30e362e4c44de98b57640746"}
Nov 02 13:11:53.963058 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:53.962669    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt" event={"ID":"64e1a02f-bf60-4321-9bd4-1313a8c4fd55","Type":"ContainerStarted","Data":"8ffc6d4ccac850fb0ff10779efcf708e99a4f9e7de2235e443869dd06ca3883e"}
Nov 02 13:11:54.080715 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:54.080665    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt" podStartSLOduration=2.080651447 podStartE2EDuration="2.080651447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:11:54.07959179 +0000 UTC m=+4542.643300199" watchObservedRunningTime="2026-11-02 13:11:54.080651447 +0000 UTC m=+4542.644359855"
Nov 02 13:11:54.528310 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:54.528233    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:11:54.969333 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:54.969301    4464 generic.go:356] "Generic (PLEG): container finished" podID="64e1a02f-bf60-4321-9bd4-1313a8c4fd55" containerID="14b159824bc37024ea1d387506b9a5e98f169b4d30e362e4c44de98b57640746" exitCode=0
Nov 02 13:11:54.969769 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:54.969388    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt" event={"ID":"64e1a02f-bf60-4321-9bd4-1313a8c4fd55","Type":"ContainerDied","Data":"14b159824bc37024ea1d387506b9a5e98f169b4d30e362e4c44de98b57640746"}
Nov 02 13:11:56.109805 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:56.109781    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"
Nov 02 13:11:56.185377 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:56.185349    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-maas-api-openshift-ca\") pod \"64e1a02f-bf60-4321-9bd4-1313a8c4fd55\" (UID: \"64e1a02f-bf60-4321-9bd4-1313a8c4fd55\") "
Nov 02 13:11:56.185568 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:56.185399    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-kube-api-access-z6x7l\" (UniqueName: \"kubernetes.io/projected/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-kube-api-access-z6x7l\") pod \"64e1a02f-bf60-4321-9bd4-1313a8c4fd55\" (UID: \"64e1a02f-bf60-4321-9bd4-1313a8c4fd55\") "
Nov 02 13:11:56.185634 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:56.185591    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-maas-api-xks-ca\") pod \"64e1a02f-bf60-4321-9bd4-1313a8c4fd55\" (UID: \"64e1a02f-bf60-4321-9bd4-1313a8c4fd55\") "
Nov 02 13:11:56.185715 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:56.185692    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-maas-api-openshift-ca" pod "64e1a02f-bf60-4321-9bd4-1313a8c4fd55" (UID: "64e1a02f-bf60-4321-9bd4-1313a8c4fd55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:11:56.185901 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:56.185881    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:11:56.187459 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:56.187438    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-kube-api-access-z6x7l" pod "64e1a02f-bf60-4321-9bd4-1313a8c4fd55" (UID: "64e1a02f-bf60-4321-9bd4-1313a8c4fd55"). InnerVolumeSpecName "kube-api-access-z6x7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:11:56.187522 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:56.187477    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-maas-api-xks-ca" pod "64e1a02f-bf60-4321-9bd4-1313a8c4fd55" (UID: "64e1a02f-bf60-4321-9bd4-1313a8c4fd55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:11:56.286463 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:56.286395    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:11:56.286463 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:56.286417    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6x7l\" (UniqueName: \"kubernetes.io/projected/64e1a02f-bf60-4321-9bd4-1313a8c4fd55-kube-api-access-z6x7l\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:11:56.982658 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:56.982632    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"
Nov 02 13:11:56.982836 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:56.982664    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt" event={"ID":"64e1a02f-bf60-4321-9bd4-1313a8c4fd55","Type":"ContainerDied","Data":"8ffc6d4ccac850fb0ff10779efcf708e99a4f9e7de2235e443869dd06ca3883e"}
Nov 02 13:11:56.982836 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:11:56.982713    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ffc6d4ccac850fb0ff10779efcf708e99a4f9e7de2235e443869dd06ca3883e"
Nov 02 13:11:58.295626 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:11:58.295590    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:12:06.221173 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.221128    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:12:06.607111 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.607024    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"]
Nov 02 13:12:06.608131 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.608105    4464 cpu_manager.go:414] "containerMap: removing container" podUID="64e1a02f-bf60-4321-9bd4-1313a8c4fd55" containerName="revoke-keys"
Nov 02 13:12:06.608301 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.608286    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e1a02f-bf60-4321-9bd4-1313a8c4fd55" containerName="revoke-keys"
Nov 02 13:12:06.608516 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.608504    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e1a02f-bf60-4321-9bd4-1313a8c4fd55" containerName="revoke-keys"
Nov 02 13:12:06.612640 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.612618    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"
Nov 02 13:12:06.685397 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.685360    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb412900-43e2-47fb-aff8-c668fd3b7cf6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95\" (UID: \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"
Nov 02 13:12:06.685552 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.685418    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkd2v\" (UniqueName: \"kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-kube-api-access-pkd2v\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95\" (UID: \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"
Nov 02 13:12:06.685552 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.685498    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb412900-43e2-47fb-aff8-c668fd3b7cf6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95\" (UID: \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"
Nov 02 13:12:06.685552 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.685533    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95\" (UID: \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"
Nov 02 13:12:06.718583 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.718549    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"]
Nov 02 13:12:06.787139 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.787110    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb412900-43e2-47fb-aff8-c668fd3b7cf6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95\" (UID: \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"
Nov 02 13:12:06.787266 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.787154    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkd2v\" (UniqueName: \"kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-kube-api-access-pkd2v\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95\" (UID: \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"
Nov 02 13:12:06.787321 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.787262    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb412900-43e2-47fb-aff8-c668fd3b7cf6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95\" (UID: \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"
Nov 02 13:12:06.787321 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.787299    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95\" (UID: \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"
Nov 02 13:12:06.787881 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.787857    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb412900-43e2-47fb-aff8-c668fd3b7cf6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95\" (UID: \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"
Nov 02 13:12:06.787971 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.787867    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb412900-43e2-47fb-aff8-c668fd3b7cf6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95\" (UID: \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"
Nov 02 13:12:06.789879 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.789858    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95\" (UID: \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"
Nov 02 13:12:06.848768 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.848740    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkd2v\" (UniqueName: \"kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-kube-api-access-pkd2v\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95\" (UID: \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"
Nov 02 13:12:06.927458 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:06.927396    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"
Nov 02 13:12:07.299374 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:07.299340    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"]
Nov 02 13:12:07.301155 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:12:07.301128    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb412900_43e2_47fb_aff8_c668fd3b7cf6.slice/crio-ef8acbbf25eda672ee2cc274bdd1384739a472cdf1c16a9ca683c735516bb7e7 WatchSource:0}: Error finding container ef8acbbf25eda672ee2cc274bdd1384739a472cdf1c16a9ca683c735516bb7e7: Status 404 returned error can't find the container with id ef8acbbf25eda672ee2cc274bdd1384739a472cdf1c16a9ca683c735516bb7e7
Nov 02 13:12:08.038214 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:08.038179    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95" event={"ID":"fb412900-43e2-47fb-aff8-c668fd3b7cf6","Type":"ContainerStarted","Data":"298d6e6dedcd975312eb5c96547e30841b9cf85b949be289a0337aef64f19533"}
Nov 02 13:12:08.038375 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:08.038243    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"
Nov 02 13:12:08.038375 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:08.038266    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95" event={"ID":"fb412900-43e2-47fb-aff8-c668fd3b7cf6","Type":"ContainerStarted","Data":"ef8acbbf25eda672ee2cc274bdd1384739a472cdf1c16a9ca683c735516bb7e7"}
Nov 02 13:12:08.156719 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:08.156670    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95" podStartSLOduration=2.156658614 podStartE2EDuration="2.156658614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:12:08.155702199 +0000 UTC m=+4556.719410619" watchObservedRunningTime="2026-11-02 13:12:08.156658614 +0000 UTC m=+4556.720367022"
Nov 02 13:12:09.109872 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:09.109837    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-4c050dcb5f96: secret "agc-cred-4c050dcb5f96" not found
Nov 02 13:12:09.109872 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:09.109868    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95: secret "agc-cred-4c050dcb5f96" not found
Nov 02 13:12:09.110458 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:09.109919    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-provider-credentials podName:fb412900-43e2-47fb-aff8-c668fd3b7cf6 nodeName:}" failed. No retries permitted until 2026-11-02 13:12:09.609903846 +0000 UTC m=+4558.173612246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95" (UID: "fb412900-43e2-47fb-aff8-c668fd3b7cf6") : secret "agc-cred-4c050dcb5f96" not found
Nov 02 13:12:09.207321 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.207281    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"]
Nov 02 13:12:09.211290 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.211267    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:09.268184 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.268150    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"]
Nov 02 13:12:09.311334 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.311304    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s\" (UID: \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:09.311565 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.311348    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsfh6\" (UniqueName: \"kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-kube-api-access-qsfh6\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s\" (UID: \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:09.311565 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.311376    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s\" (UID: \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:09.311565 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.311518    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s\" (UID: \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:09.412382 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.412308    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s\" (UID: \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:09.412382 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.412351    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s\" (UID: \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:09.412613 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.412450    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s\" (UID: \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:09.412613 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:09.412477    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-2a587a2bd360: secret "agc-cred-2a587a2bd360" not found
Nov 02 13:12:09.412613 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:09.412503    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s: secret "agc-cred-2a587a2bd360" not found
Nov 02 13:12:09.412613 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.412550    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsfh6\" (UniqueName: \"kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-kube-api-access-qsfh6\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s\" (UID: \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:09.412613 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:09.412572    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-provider-credentials podName:a6e6e343-816c-4fe8-ab37-ae0ad3553c40 nodeName:}" failed. No retries permitted until 2026-11-02 13:12:09.912551696 +0000 UTC m=+4558.476260106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s" (UID: "a6e6e343-816c-4fe8-ab37-ae0ad3553c40") : secret "agc-cred-2a587a2bd360" not found
Nov 02 13:12:09.413103 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.413084    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s\" (UID: \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:09.413197 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.413180    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s\" (UID: \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:09.474400 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.474375    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsfh6\" (UniqueName: \"kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-kube-api-access-qsfh6\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s\" (UID: \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:09.614602 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:09.614569    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-4c050dcb5f96: secret "agc-cred-4c050dcb5f96" not found
Nov 02 13:12:09.614602 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:09.614597    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95: secret "agc-cred-4c050dcb5f96" not found
Nov 02 13:12:09.614860 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:09.614676    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-provider-credentials podName:fb412900-43e2-47fb-aff8-c668fd3b7cf6 nodeName:}" failed. No retries permitted until 2026-11-02 13:12:10.614657602 +0000 UTC m=+4559.178366003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95" (UID: "fb412900-43e2-47fb-aff8-c668fd3b7cf6") : secret "agc-cred-4c050dcb5f96" not found
Nov 02 13:12:09.832437 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.832408    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"]
Nov 02 13:12:09.836190 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.836171    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"
Nov 02 13:12:09.889926 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.889896    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:12:09.890164 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.890102    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 13:12:09.890281 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.890232    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:12:09.890356 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.890290    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:12:09.892733 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.892708    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"]
Nov 02 13:12:09.916846 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.916816    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s\" (UID: \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:09.916965 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.916869    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w24gb\" (UniqueName: \"kubernetes.io/projected/17410b5d-cf59-4ef2-90f9-e10d8bea329f-kube-api-access-w24gb\") pod \"maas-api-revoke-sub-5af16306-dmbvs\" (UID: \"17410b5d-cf59-4ef2-90f9-e10d8bea329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"
Nov 02 13:12:09.916965 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.916952    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17410b5d-cf59-4ef2-90f9-e10d8bea329f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5af16306-dmbvs\" (UID: \"17410b5d-cf59-4ef2-90f9-e10d8bea329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"
Nov 02 13:12:09.917070 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:09.916995    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-2a587a2bd360: secret "agc-cred-2a587a2bd360" not found
Nov 02 13:12:09.917070 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:09.917011    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s: secret "agc-cred-2a587a2bd360" not found
Nov 02 13:12:09.917152 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:09.917081    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-provider-credentials podName:a6e6e343-816c-4fe8-ab37-ae0ad3553c40 nodeName:}" failed. No retries permitted until 2026-11-02 13:12:10.917064661 +0000 UTC m=+4559.480773068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s" (UID: "a6e6e343-816c-4fe8-ab37-ae0ad3553c40") : secret "agc-cred-2a587a2bd360" not found
Nov 02 13:12:09.917262 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:09.917240    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17410b5d-cf59-4ef2-90f9-e10d8bea329f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5af16306-dmbvs\" (UID: \"17410b5d-cf59-4ef2-90f9-e10d8bea329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"
Nov 02 13:12:10.018445 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:10.018413    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17410b5d-cf59-4ef2-90f9-e10d8bea329f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5af16306-dmbvs\" (UID: \"17410b5d-cf59-4ef2-90f9-e10d8bea329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"
Nov 02 13:12:10.018638 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:10.018505    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17410b5d-cf59-4ef2-90f9-e10d8bea329f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5af16306-dmbvs\" (UID: \"17410b5d-cf59-4ef2-90f9-e10d8bea329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"
Nov 02 13:12:10.018638 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:10.018549    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w24gb\" (UniqueName: \"kubernetes.io/projected/17410b5d-cf59-4ef2-90f9-e10d8bea329f-kube-api-access-w24gb\") pod \"maas-api-revoke-sub-5af16306-dmbvs\" (UID: \"17410b5d-cf59-4ef2-90f9-e10d8bea329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"
Nov 02 13:12:10.019156 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:10.019131    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17410b5d-cf59-4ef2-90f9-e10d8bea329f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5af16306-dmbvs\" (UID: \"17410b5d-cf59-4ef2-90f9-e10d8bea329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"
Nov 02 13:12:10.020803 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:10.020782    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17410b5d-cf59-4ef2-90f9-e10d8bea329f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5af16306-dmbvs\" (UID: \"17410b5d-cf59-4ef2-90f9-e10d8bea329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"
Nov 02 13:12:10.080633 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:10.080602    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w24gb\" (UniqueName: \"kubernetes.io/projected/17410b5d-cf59-4ef2-90f9-e10d8bea329f-kube-api-access-w24gb\") pod \"maas-api-revoke-sub-5af16306-dmbvs\" (UID: \"17410b5d-cf59-4ef2-90f9-e10d8bea329f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"
Nov 02 13:12:10.149445 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:10.149366    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"
Nov 02 13:12:10.522719 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:10.522689    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"]
Nov 02 13:12:10.524465 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:12:10.524430    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17410b5d_cf59_4ef2_90f9_e10d8bea329f.slice/crio-e5e4b1ae4b8d9940b81659f8185736142a90d3d608f6ec302f0434a02e3b8a6a WatchSource:0}: Error finding container e5e4b1ae4b8d9940b81659f8185736142a90d3d608f6ec302f0434a02e3b8a6a: Status 404 returned error can't find the container with id e5e4b1ae4b8d9940b81659f8185736142a90d3d608f6ec302f0434a02e3b8a6a
Nov 02 13:12:10.623650 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:10.623621    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-4c050dcb5f96: secret "agc-cred-4c050dcb5f96" not found
Nov 02 13:12:10.623650 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:10.623649    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95: secret "agc-cred-4c050dcb5f96" not found
Nov 02 13:12:10.623857 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:10.623698    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-provider-credentials podName:fb412900-43e2-47fb-aff8-c668fd3b7cf6 nodeName:}" failed. No retries permitted until 2026-11-02 13:12:12.623685617 +0000 UTC m=+4561.187394005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95" (UID: "fb412900-43e2-47fb-aff8-c668fd3b7cf6") : secret "agc-cred-4c050dcb5f96" not found
Nov 02 13:12:10.927205 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:10.927170    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s\" (UID: \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:10.927352 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:10.927314    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-2a587a2bd360: secret "agc-cred-2a587a2bd360" not found
Nov 02 13:12:10.927352 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:10.927334    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s: secret "agc-cred-2a587a2bd360" not found
Nov 02 13:12:10.927447 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:10.927381    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-provider-credentials podName:a6e6e343-816c-4fe8-ab37-ae0ad3553c40 nodeName:}" failed. No retries permitted until 2026-11-02 13:12:12.927367204 +0000 UTC m=+4561.491075608 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s" (UID: "a6e6e343-816c-4fe8-ab37-ae0ad3553c40") : secret "agc-cred-2a587a2bd360" not found
Nov 02 13:12:11.054496 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:11.054411    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs" event={"ID":"17410b5d-cf59-4ef2-90f9-e10d8bea329f","Type":"ContainerStarted","Data":"326d1632c6d93644ef7ed4624897af7a710f9898f25c6a502a007bbffe8d4992"}
Nov 02 13:12:11.054496 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:11.054476    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs" event={"ID":"17410b5d-cf59-4ef2-90f9-e10d8bea329f","Type":"ContainerStarted","Data":"e5e4b1ae4b8d9940b81659f8185736142a90d3d608f6ec302f0434a02e3b8a6a"}
Nov 02 13:12:11.172263 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:11.172214    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs" podStartSLOduration=2.172199301 podStartE2EDuration="2.172199301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:12:11.17116146 +0000 UTC m=+4559.734869874" watchObservedRunningTime="2026-11-02 13:12:11.172199301 +0000 UTC m=+4559.735907710"
Nov 02 13:12:11.793294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:11.793257    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"]
Nov 02 13:12:11.820448 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:11.820412    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"]
Nov 02 13:12:11.824168 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:11.824144    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"
Nov 02 13:12:11.906161 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:11.906125    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"]
Nov 02 13:12:11.937403 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:11.937371    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4773433e-d8c6-4bb0-b9e9-00dadae397ca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z\" (UID: \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"
Nov 02 13:12:11.937552 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:11.937436    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4773433e-d8c6-4bb0-b9e9-00dadae397ca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z\" (UID: \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"
Nov 02 13:12:11.937552 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:11.937539    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqhrg\" (UniqueName: \"kubernetes.io/projected/4773433e-d8c6-4bb0-b9e9-00dadae397ca-kube-api-access-xqhrg\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z\" (UID: \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"
Nov 02 13:12:11.937652 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:11.937564    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4773433e-d8c6-4bb0-b9e9-00dadae397ca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z\" (UID: \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"
Nov 02 13:12:12.038596 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:12.038559    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4773433e-d8c6-4bb0-b9e9-00dadae397ca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z\" (UID: \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"
Nov 02 13:12:12.038808 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:12.038699    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4773433e-d8c6-4bb0-b9e9-00dadae397ca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z\" (UID: \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"
Nov 02 13:12:12.038808 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:12.038761    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4773433e-d8c6-4bb0-b9e9-00dadae397ca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z\" (UID: \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"
Nov 02 13:12:12.038908 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:12.038852    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqhrg\" (UniqueName: \"kubernetes.io/projected/4773433e-d8c6-4bb0-b9e9-00dadae397ca-kube-api-access-xqhrg\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z\" (UID: \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"
Nov 02 13:12:12.039389 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:12.039364    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4773433e-d8c6-4bb0-b9e9-00dadae397ca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z\" (UID: \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"
Nov 02 13:12:12.039510 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:12.039473    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4773433e-d8c6-4bb0-b9e9-00dadae397ca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z\" (UID: \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"
Nov 02 13:12:12.041344 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:12.041320    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4773433e-d8c6-4bb0-b9e9-00dadae397ca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z\" (UID: \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"
Nov 02 13:12:12.061073 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:12.061004    4464 generic.go:356] "Generic (PLEG): container finished" podID="17410b5d-cf59-4ef2-90f9-e10d8bea329f" containerID="326d1632c6d93644ef7ed4624897af7a710f9898f25c6a502a007bbffe8d4992" exitCode=0
Nov 02 13:12:12.061073 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:12.061038    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs" event={"ID":"17410b5d-cf59-4ef2-90f9-e10d8bea329f","Type":"ContainerDied","Data":"326d1632c6d93644ef7ed4624897af7a710f9898f25c6a502a007bbffe8d4992"}
Nov 02 13:12:12.100962 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:12.100935    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqhrg\" (UniqueName: \"kubernetes.io/projected/4773433e-d8c6-4bb0-b9e9-00dadae397ca-kube-api-access-xqhrg\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z\" (UID: \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"
Nov 02 13:12:12.137484 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:12.137456    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"
Nov 02 13:12:12.513939 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:12.513910    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"]
Nov 02 13:12:12.515434 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:12:12.515403    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4773433e_d8c6_4bb0_b9e9_00dadae397ca.slice/crio-727934eeb7fa7d9ae06beca9be77ca153e2562207f4e7799de715c35d8034745 WatchSource:0}: Error finding container 727934eeb7fa7d9ae06beca9be77ca153e2562207f4e7799de715c35d8034745: Status 404 returned error can't find the container with id 727934eeb7fa7d9ae06beca9be77ca153e2562207f4e7799de715c35d8034745
Nov 02 13:12:12.645176 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:12.645149    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-4c050dcb5f96: secret "agc-cred-4c050dcb5f96" not found
Nov 02 13:12:12.645176 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:12.645172    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95: secret "agc-cred-4c050dcb5f96" not found
Nov 02 13:12:12.645375 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:12.645231    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-provider-credentials podName:fb412900-43e2-47fb-aff8-c668fd3b7cf6 nodeName:}" failed. No retries permitted until 2026-11-02 13:12:16.645216511 +0000 UTC m=+4565.208924919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95" (UID: "fb412900-43e2-47fb-aff8-c668fd3b7cf6") : secret "agc-cred-4c050dcb5f96" not found
Nov 02 13:12:12.947385 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:12.947298    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s\" (UID: \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:12.947536 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:12.947448    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-2a587a2bd360: secret "agc-cred-2a587a2bd360" not found
Nov 02 13:12:12.947536 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:12.947471    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s: secret "agc-cred-2a587a2bd360" not found
Nov 02 13:12:12.947536 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:12.947523    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-provider-credentials podName:a6e6e343-816c-4fe8-ab37-ae0ad3553c40 nodeName:}" failed. No retries permitted until 2026-11-02 13:12:16.94750924 +0000 UTC m=+4565.511217631 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s" (UID: "a6e6e343-816c-4fe8-ab37-ae0ad3553c40") : secret "agc-cred-2a587a2bd360" not found
Nov 02 13:12:13.068364 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:13.068330    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z" event={"ID":"4773433e-d8c6-4bb0-b9e9-00dadae397ca","Type":"ContainerStarted","Data":"ebeea8615322436d12c5dfaa48a7b6876c9c564a0ebd2e47c682a46f75096ca3"}
Nov 02 13:12:13.068531 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:13.068381    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z" event={"ID":"4773433e-d8c6-4bb0-b9e9-00dadae397ca","Type":"ContainerStarted","Data":"727934eeb7fa7d9ae06beca9be77ca153e2562207f4e7799de715c35d8034745"}
Nov 02 13:12:13.068531 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:13.068445    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"
Nov 02 13:12:13.188211 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:13.188164    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z" podStartSLOduration=2.1881519799999998 podStartE2EDuration="2.18815198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:12:13.186487266 +0000 UTC m=+4561.750195674" watchObservedRunningTime="2026-11-02 13:12:13.18815198 +0000 UTC m=+4561.751860389"
Nov 02 13:12:13.208969 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:13.208900    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"
Nov 02 13:12:13.350757 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:13.350727    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17410b5d-cf59-4ef2-90f9-e10d8bea329f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17410b5d-cf59-4ef2-90f9-e10d8bea329f-maas-api-openshift-ca\") pod \"17410b5d-cf59-4ef2-90f9-e10d8bea329f\" (UID: \"17410b5d-cf59-4ef2-90f9-e10d8bea329f\") "
Nov 02 13:12:13.351004 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:13.350817    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17410b5d-cf59-4ef2-90f9-e10d8bea329f-kube-api-access-w24gb\" (UniqueName: \"kubernetes.io/projected/17410b5d-cf59-4ef2-90f9-e10d8bea329f-kube-api-access-w24gb\") pod \"17410b5d-cf59-4ef2-90f9-e10d8bea329f\" (UID: \"17410b5d-cf59-4ef2-90f9-e10d8bea329f\") "
Nov 02 13:12:13.351004 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:13.350963    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17410b5d-cf59-4ef2-90f9-e10d8bea329f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17410b5d-cf59-4ef2-90f9-e10d8bea329f-maas-api-xks-ca\") pod \"17410b5d-cf59-4ef2-90f9-e10d8bea329f\" (UID: \"17410b5d-cf59-4ef2-90f9-e10d8bea329f\") "
Nov 02 13:12:13.351173 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:13.351133    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17410b5d-cf59-4ef2-90f9-e10d8bea329f-maas-api-openshift-ca" pod "17410b5d-cf59-4ef2-90f9-e10d8bea329f" (UID: "17410b5d-cf59-4ef2-90f9-e10d8bea329f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:12:13.351361 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:13.351341    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17410b5d-cf59-4ef2-90f9-e10d8bea329f-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:13.353034 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:13.352998    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17410b5d-cf59-4ef2-90f9-e10d8bea329f-maas-api-xks-ca" pod "17410b5d-cf59-4ef2-90f9-e10d8bea329f" (UID: "17410b5d-cf59-4ef2-90f9-e10d8bea329f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:12:13.353034 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:13.353018    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17410b5d-cf59-4ef2-90f9-e10d8bea329f-kube-api-access-w24gb" pod "17410b5d-cf59-4ef2-90f9-e10d8bea329f" (UID: "17410b5d-cf59-4ef2-90f9-e10d8bea329f"). InnerVolumeSpecName "kube-api-access-w24gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:12:13.452900 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:13.452866    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17410b5d-cf59-4ef2-90f9-e10d8bea329f-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:13.452900 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:13.452894    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w24gb\" (UniqueName: \"kubernetes.io/projected/17410b5d-cf59-4ef2-90f9-e10d8bea329f-kube-api-access-w24gb\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:14.076874 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.076837    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"
Nov 02 13:12:14.076874 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.076846    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs" event={"ID":"17410b5d-cf59-4ef2-90f9-e10d8bea329f","Type":"ContainerDied","Data":"e5e4b1ae4b8d9940b81659f8185736142a90d3d608f6ec302f0434a02e3b8a6a"}
Nov 02 13:12:14.077341 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.076914    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5e4b1ae4b8d9940b81659f8185736142a90d3d608f6ec302f0434a02e3b8a6a"
Nov 02 13:12:14.402853 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.402768    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"]
Nov 02 13:12:14.403056 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:14.403037    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s" podUID="a6e6e343-816c-4fe8-ab37-ae0ad3553c40"
Nov 02 13:12:14.434778 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.434748    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"]
Nov 02 13:12:14.435270 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.435255    4464 cpu_manager.go:414] "containerMap: removing container" podUID="17410b5d-cf59-4ef2-90f9-e10d8bea329f" containerName="revoke-keys"
Nov 02 13:12:14.435348 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.435271    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17410b5d-cf59-4ef2-90f9-e10d8bea329f" containerName="revoke-keys"
Nov 02 13:12:14.435383 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.435365    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="17410b5d-cf59-4ef2-90f9-e10d8bea329f" containerName="revoke-keys"
Nov 02 13:12:14.438760 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.438746    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"
Nov 02 13:12:14.498793 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.498760    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"]
Nov 02 13:12:14.565134 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.565093    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7ce9975-1859-40b9-933b-a9f52618008f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv\" (UID: \"b7ce9975-1859-40b9-933b-a9f52618008f\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"
Nov 02 13:12:14.565134 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.565140    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7ce9975-1859-40b9-933b-a9f52618008f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv\" (UID: \"b7ce9975-1859-40b9-933b-a9f52618008f\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"
Nov 02 13:12:14.565372 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.565255    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7ce9975-1859-40b9-933b-a9f52618008f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv\" (UID: \"b7ce9975-1859-40b9-933b-a9f52618008f\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"
Nov 02 13:12:14.565372 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.565358    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87btw\" (UniqueName: \"kubernetes.io/projected/b7ce9975-1859-40b9-933b-a9f52618008f-kube-api-access-87btw\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv\" (UID: \"b7ce9975-1859-40b9-933b-a9f52618008f\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"
Nov 02 13:12:14.666594 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.666509    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87btw\" (UniqueName: \"kubernetes.io/projected/b7ce9975-1859-40b9-933b-a9f52618008f-kube-api-access-87btw\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv\" (UID: \"b7ce9975-1859-40b9-933b-a9f52618008f\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"
Nov 02 13:12:14.666594 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.666588    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7ce9975-1859-40b9-933b-a9f52618008f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv\" (UID: \"b7ce9975-1859-40b9-933b-a9f52618008f\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"
Nov 02 13:12:14.666824 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.666623    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7ce9975-1859-40b9-933b-a9f52618008f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv\" (UID: \"b7ce9975-1859-40b9-933b-a9f52618008f\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"
Nov 02 13:12:14.666824 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.666677    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7ce9975-1859-40b9-933b-a9f52618008f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv\" (UID: \"b7ce9975-1859-40b9-933b-a9f52618008f\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"
Nov 02 13:12:14.667430 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.667405    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7ce9975-1859-40b9-933b-a9f52618008f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv\" (UID: \"b7ce9975-1859-40b9-933b-a9f52618008f\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"
Nov 02 13:12:14.667537 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.667432    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7ce9975-1859-40b9-933b-a9f52618008f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv\" (UID: \"b7ce9975-1859-40b9-933b-a9f52618008f\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"
Nov 02 13:12:14.669009 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.668992    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7ce9975-1859-40b9-933b-a9f52618008f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv\" (UID: \"b7ce9975-1859-40b9-933b-a9f52618008f\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"
Nov 02 13:12:14.728263 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.728229    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87btw\" (UniqueName: \"kubernetes.io/projected/b7ce9975-1859-40b9-933b-a9f52618008f-kube-api-access-87btw\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv\" (UID: \"b7ce9975-1859-40b9-933b-a9f52618008f\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"
Nov 02 13:12:14.752215 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:14.752173    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"
Nov 02 13:12:15.081574 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:15.081542    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:15.089942 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:15.089918    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:15.124534 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:15.124499    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"]
Nov 02 13:12:15.124778 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:12:15.124757    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ce9975_1859_40b9_933b_a9f52618008f.slice/crio-ec9f40a0d057c6b85272428ed174cc8ad5c2188f2cf5aab38165e58504ae0fca WatchSource:0}: Error finding container ec9f40a0d057c6b85272428ed174cc8ad5c2188f2cf5aab38165e58504ae0fca: Status 404 returned error can't find the container with id ec9f40a0d057c6b85272428ed174cc8ad5c2188f2cf5aab38165e58504ae0fca
Nov 02 13:12:15.171607 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:15.171584    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-kube-api-access-qsfh6\" (UniqueName: \"kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-kube-api-access-qsfh6\") pod \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\" (UID: \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\") "
Nov 02 13:12:15.171743 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:15.171632    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-plugins-config-volume\") pod \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\" (UID: \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\") "
Nov 02 13:12:15.171743 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:15.171673    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-routing-overlay-volume\") pod \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\" (UID: \"a6e6e343-816c-4fe8-ab37-ae0ad3553c40\") "
Nov 02 13:12:15.172108 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:15.172085    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-plugins-config-volume" pod "a6e6e343-816c-4fe8-ab37-ae0ad3553c40" (UID: "a6e6e343-816c-4fe8-ab37-ae0ad3553c40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:12:15.172108 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:15.172098    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-routing-overlay-volume" pod "a6e6e343-816c-4fe8-ab37-ae0ad3553c40" (UID: "a6e6e343-816c-4fe8-ab37-ae0ad3553c40"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:12:15.173853 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:15.173827    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-kube-api-access-qsfh6" pod "a6e6e343-816c-4fe8-ab37-ae0ad3553c40" (UID: "a6e6e343-816c-4fe8-ab37-ae0ad3553c40"). InnerVolumeSpecName "kube-api-access-qsfh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:12:15.273476 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:15.273438    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsfh6\" (UniqueName: \"kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-kube-api-access-qsfh6\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:15.273476 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:15.273467    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:15.273476 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:15.273479    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:16.087997 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:16.087941    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv" event={"ID":"b7ce9975-1859-40b9-933b-a9f52618008f","Type":"ContainerStarted","Data":"6c02b2a7b749d6cc6d00f58f4920431e21d4d91e734762ee6bec44b21c0748c2"}
Nov 02 13:12:16.088425 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:16.088023    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv" event={"ID":"b7ce9975-1859-40b9-933b-a9f52618008f","Type":"ContainerStarted","Data":"ec9f40a0d057c6b85272428ed174cc8ad5c2188f2cf5aab38165e58504ae0fca"}
Nov 02 13:12:16.088425 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:16.088029    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"
Nov 02 13:12:16.088425 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:16.088053    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"
Nov 02 13:12:16.206852 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:16.206806    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv" podStartSLOduration=2.206794435 podStartE2EDuration="2.206794435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:12:16.206320674 +0000 UTC m=+4564.770029085" watchObservedRunningTime="2026-11-02 13:12:16.206794435 +0000 UTC m=+4564.770502844"
Nov 02 13:12:16.383543 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:16.383458    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"]
Nov 02 13:12:16.387422 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:16.387395    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-69fd77b4s"]
Nov 02 13:12:16.485314 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:16.485281    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6e6e343-816c-4fe8-ab37-ae0ad3553c40-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:16.687246 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:16.687159    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-4c050dcb5f96: secret "agc-cred-4c050dcb5f96" not found
Nov 02 13:12:16.687246 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:16.687185    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95: secret "agc-cred-4c050dcb5f96" not found
Nov 02 13:12:16.687246 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:16.687239    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-provider-credentials podName:fb412900-43e2-47fb-aff8-c668fd3b7cf6 nodeName:}" failed. No retries permitted until 2026-11-02 13:12:24.687224779 +0000 UTC m=+4573.250933166 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95" (UID: "fb412900-43e2-47fb-aff8-c668fd3b7cf6") : secret "agc-cred-4c050dcb5f96" not found
Nov 02 13:12:16.794282 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:16.794243    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95" podUID="fb412900-43e2-47fb-aff8-c668fd3b7cf6" containerName="payload-processing" containerID="cri-o://298d6e6dedcd975312eb5c96547e30841b9cf85b949be289a0337aef64f19533" gracePeriod=25
Nov 02 13:12:17.044393 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.044370    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"
Nov 02 13:12:17.093606 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.093578    4464 generic.go:356] "Generic (PLEG): container finished" podID="fb412900-43e2-47fb-aff8-c668fd3b7cf6" containerID="298d6e6dedcd975312eb5c96547e30841b9cf85b949be289a0337aef64f19533" exitCode=0
Nov 02 13:12:17.094010 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.093626    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"
Nov 02 13:12:17.094010 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.093671    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95" event={"ID":"fb412900-43e2-47fb-aff8-c668fd3b7cf6","Type":"ContainerDied","Data":"298d6e6dedcd975312eb5c96547e30841b9cf85b949be289a0337aef64f19533"}
Nov 02 13:12:17.094010 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.093725    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95" event={"ID":"fb412900-43e2-47fb-aff8-c668fd3b7cf6","Type":"ContainerDied","Data":"ef8acbbf25eda672ee2cc274bdd1384739a472cdf1c16a9ca683c735516bb7e7"}
Nov 02 13:12:17.094010 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.093748    4464 scope.go:122] "RemoveContainer" containerID="298d6e6dedcd975312eb5c96547e30841b9cf85b949be289a0337aef64f19533"
Nov 02 13:12:17.102380 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.102364    4464 scope.go:122] "RemoveContainer" containerID="298d6e6dedcd975312eb5c96547e30841b9cf85b949be289a0337aef64f19533"
Nov 02 13:12:17.102645 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:17.102619    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298d6e6dedcd975312eb5c96547e30841b9cf85b949be289a0337aef64f19533\": container with ID starting with 298d6e6dedcd975312eb5c96547e30841b9cf85b949be289a0337aef64f19533 not found: ID does not exist" containerID="298d6e6dedcd975312eb5c96547e30841b9cf85b949be289a0337aef64f19533"
Nov 02 13:12:17.102691 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.102652    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298d6e6dedcd975312eb5c96547e30841b9cf85b949be289a0337aef64f19533"} err="failed to get container status \"298d6e6dedcd975312eb5c96547e30841b9cf85b949be289a0337aef64f19533\": rpc error: code = NotFound desc = could not find container \"298d6e6dedcd975312eb5c96547e30841b9cf85b949be289a0337aef64f19533\": container with ID starting with 298d6e6dedcd975312eb5c96547e30841b9cf85b949be289a0337aef64f19533 not found: ID does not exist"
Nov 02 13:12:17.190278 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.190246    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb412900-43e2-47fb-aff8-c668fd3b7cf6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb412900-43e2-47fb-aff8-c668fd3b7cf6-routing-overlay-volume\") pod \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\" (UID: \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\") "
Nov 02 13:12:17.190453 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.190299    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-kube-api-access-pkd2v\" (UniqueName: \"kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-kube-api-access-pkd2v\") pod \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\" (UID: \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\") "
Nov 02 13:12:17.190520 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.190458    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb412900-43e2-47fb-aff8-c668fd3b7cf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb412900-43e2-47fb-aff8-c668fd3b7cf6-plugins-config-volume\") pod \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\" (UID: \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\") "
Nov 02 13:12:17.190520 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.190499    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-provider-credentials\") pod \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\" (UID: \"fb412900-43e2-47fb-aff8-c668fd3b7cf6\") "
Nov 02 13:12:17.190754 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.190662    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb412900-43e2-47fb-aff8-c668fd3b7cf6-routing-overlay-volume" pod "fb412900-43e2-47fb-aff8-c668fd3b7cf6" (UID: "fb412900-43e2-47fb-aff8-c668fd3b7cf6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:12:17.190918 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.190884    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb412900-43e2-47fb-aff8-c668fd3b7cf6-plugins-config-volume" pod "fb412900-43e2-47fb-aff8-c668fd3b7cf6" (UID: "fb412900-43e2-47fb-aff8-c668fd3b7cf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:12:17.191049 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.191025    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb412900-43e2-47fb-aff8-c668fd3b7cf6-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:17.191160 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.191052    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb412900-43e2-47fb-aff8-c668fd3b7cf6-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:17.192545 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.192526    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-provider-credentials" pod "fb412900-43e2-47fb-aff8-c668fd3b7cf6" (UID: "fb412900-43e2-47fb-aff8-c668fd3b7cf6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:12:17.192630 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.192610    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-kube-api-access-pkd2v" pod "fb412900-43e2-47fb-aff8-c668fd3b7cf6" (UID: "fb412900-43e2-47fb-aff8-c668fd3b7cf6"). InnerVolumeSpecName "kube-api-access-pkd2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:12:17.292613 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.292579    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkd2v\" (UniqueName: \"kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-kube-api-access-pkd2v\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:17.292613 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.292608    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb412900-43e2-47fb-aff8-c668fd3b7cf6-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:17.581531 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.581456    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e6e343-816c-4fe8-ab37-ae0ad3553c40" path="/var/lib/kubelet/pods/a6e6e343-816c-4fe8-ab37-ae0ad3553c40/volumes"
Nov 02 13:12:17.587435 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.587405    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"]
Nov 02 13:12:17.638183 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:17.638157    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-5f84ffs95"]
Nov 02 13:12:19.581630 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:19.581600    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb412900-43e2-47fb-aff8-c668fd3b7cf6" path="/var/lib/kubelet/pods/fb412900-43e2-47fb-aff8-c668fd3b7cf6/volumes"
Nov 02 13:12:19.983725 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:19.983694    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:12:21.339800 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.339767    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"]
Nov 02 13:12:21.340491 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.340470    4464 cpu_manager.go:414] "containerMap: removing container" podUID="fb412900-43e2-47fb-aff8-c668fd3b7cf6" containerName="payload-processing"
Nov 02 13:12:21.340595 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.340493    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb412900-43e2-47fb-aff8-c668fd3b7cf6" containerName="payload-processing"
Nov 02 13:12:21.340660 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.340632    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb412900-43e2-47fb-aff8-c668fd3b7cf6" containerName="payload-processing"
Nov 02 13:12:21.345264 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.345245    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"
Nov 02 13:12:21.401553 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.401520    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"]
Nov 02 13:12:21.434221 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.434195    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smvl8\" (UniqueName: \"kubernetes.io/projected/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-kube-api-access-smvl8\") pod \"maas-api-revoke-sub-e59e625b-ggdjv\" (UID: \"bae14ab2-1a37-440e-ad0b-0fb57a7ef755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"
Nov 02 13:12:21.434372 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.434227    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59e625b-ggdjv\" (UID: \"bae14ab2-1a37-440e-ad0b-0fb57a7ef755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"
Nov 02 13:12:21.434372 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.434250    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59e625b-ggdjv\" (UID: \"bae14ab2-1a37-440e-ad0b-0fb57a7ef755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"
Nov 02 13:12:21.448845 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.448819    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:12:21.448845 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.448822    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:12:21.449018 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.448879    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:12:21.449018 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.448952    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 13:12:21.535477 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.535447    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smvl8\" (UniqueName: \"kubernetes.io/projected/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-kube-api-access-smvl8\") pod \"maas-api-revoke-sub-e59e625b-ggdjv\" (UID: \"bae14ab2-1a37-440e-ad0b-0fb57a7ef755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"
Nov 02 13:12:21.535477 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.535480    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59e625b-ggdjv\" (UID: \"bae14ab2-1a37-440e-ad0b-0fb57a7ef755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"
Nov 02 13:12:21.535696 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.535501    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59e625b-ggdjv\" (UID: \"bae14ab2-1a37-440e-ad0b-0fb57a7ef755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"
Nov 02 13:12:21.536203 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.536183    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59e625b-ggdjv\" (UID: \"bae14ab2-1a37-440e-ad0b-0fb57a7ef755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"
Nov 02 13:12:21.537774 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.537745    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59e625b-ggdjv\" (UID: \"bae14ab2-1a37-440e-ad0b-0fb57a7ef755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"
Nov 02 13:12:21.597302 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.597235    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smvl8\" (UniqueName: \"kubernetes.io/projected/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-kube-api-access-smvl8\") pod \"maas-api-revoke-sub-e59e625b-ggdjv\" (UID: \"bae14ab2-1a37-440e-ad0b-0fb57a7ef755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"
Nov 02 13:12:21.658032 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:21.658011    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"
Nov 02 13:12:22.029698 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:22.029671    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"]
Nov 02 13:12:22.032600 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:12:22.032572    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae14ab2_1a37_440e_ad0b_0fb57a7ef755.slice/crio-de0576180379e928bcef14d2da59cf77d0d630c0b5703d0dad7d6c6821bf10c0 WatchSource:0}: Error finding container de0576180379e928bcef14d2da59cf77d0d630c0b5703d0dad7d6c6821bf10c0: Status 404 returned error can't find the container with id de0576180379e928bcef14d2da59cf77d0d630c0b5703d0dad7d6c6821bf10c0
Nov 02 13:12:22.121581 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:22.121555    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv" event={"ID":"bae14ab2-1a37-440e-ad0b-0fb57a7ef755","Type":"ContainerStarted","Data":"de0576180379e928bcef14d2da59cf77d0d630c0b5703d0dad7d6c6821bf10c0"}
Nov 02 13:12:23.127948 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:23.127865    4464 generic.go:356] "Generic (PLEG): container finished" podID="bae14ab2-1a37-440e-ad0b-0fb57a7ef755" containerID="3fd88571d203d0f3febd9e02525eaff4acc4d4f8e8f40005b5b9f639ba89ac93" exitCode=0
Nov 02 13:12:23.127948 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:23.127929    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv" event={"ID":"bae14ab2-1a37-440e-ad0b-0fb57a7ef755","Type":"ContainerDied","Data":"3fd88571d203d0f3febd9e02525eaff4acc4d4f8e8f40005b5b9f639ba89ac93"}
Nov 02 13:12:24.080540 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:24.080505    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z" podUID="4773433e-d8c6-4bb0-b9e9-00dadae397ca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:12:24.262126 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:24.262100    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"
Nov 02 13:12:24.361285 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:24.361208    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-kube-api-access-smvl8\" (UniqueName: \"kubernetes.io/projected/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-kube-api-access-smvl8\") pod \"bae14ab2-1a37-440e-ad0b-0fb57a7ef755\" (UID: \"bae14ab2-1a37-440e-ad0b-0fb57a7ef755\") "
Nov 02 13:12:24.361285 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:24.361244    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-maas-api-xks-ca\") pod \"bae14ab2-1a37-440e-ad0b-0fb57a7ef755\" (UID: \"bae14ab2-1a37-440e-ad0b-0fb57a7ef755\") "
Nov 02 13:12:24.361507 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:24.361428    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-maas-api-openshift-ca\") pod \"bae14ab2-1a37-440e-ad0b-0fb57a7ef755\" (UID: \"bae14ab2-1a37-440e-ad0b-0fb57a7ef755\") "
Nov 02 13:12:24.361761 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:24.361739    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-maas-api-openshift-ca" pod "bae14ab2-1a37-440e-ad0b-0fb57a7ef755" (UID: "bae14ab2-1a37-440e-ad0b-0fb57a7ef755"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:12:24.361909 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:24.361895    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:24.363235 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:24.363211    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-maas-api-xks-ca" pod "bae14ab2-1a37-440e-ad0b-0fb57a7ef755" (UID: "bae14ab2-1a37-440e-ad0b-0fb57a7ef755"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:12:24.363324 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:24.363267    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-kube-api-access-smvl8" pod "bae14ab2-1a37-440e-ad0b-0fb57a7ef755" (UID: "bae14ab2-1a37-440e-ad0b-0fb57a7ef755"). InnerVolumeSpecName "kube-api-access-smvl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:12:24.462558 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:24.462527    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smvl8\" (UniqueName: \"kubernetes.io/projected/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-kube-api-access-smvl8\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:24.462558 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:24.462552    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae14ab2-1a37-440e-ad0b-0fb57a7ef755-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:25.140683 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:25.140652    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"
Nov 02 13:12:25.140840 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:25.140687    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv" event={"ID":"bae14ab2-1a37-440e-ad0b-0fb57a7ef755","Type":"ContainerDied","Data":"de0576180379e928bcef14d2da59cf77d0d630c0b5703d0dad7d6c6821bf10c0"}
Nov 02 13:12:25.140840 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:25.140737    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de0576180379e928bcef14d2da59cf77d0d630c0b5703d0dad7d6c6821bf10c0"
Nov 02 13:12:27.096989 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:27.096931    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv" podUID="b7ce9975-1859-40b9-933b-a9f52618008f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:12:28.291303 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:28.291269    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:12:30.595103 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:30.595069    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:12:34.079099 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:34.079058    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z" podUID="4773433e-d8c6-4bb0-b9e9-00dadae397ca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:12:35.077387 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.077355    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"]
Nov 02 13:12:35.077929 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.077908    4464 cpu_manager.go:414] "containerMap: removing container" podUID="bae14ab2-1a37-440e-ad0b-0fb57a7ef755" containerName="revoke-keys"
Nov 02 13:12:35.077929 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.077928    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae14ab2-1a37-440e-ad0b-0fb57a7ef755" containerName="revoke-keys"
Nov 02 13:12:35.078115 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.078030    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae14ab2-1a37-440e-ad0b-0fb57a7ef755" containerName="revoke-keys"
Nov 02 13:12:35.081272 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.081257    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"
Nov 02 13:12:35.134609 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.134552    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:12:35.134609 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.134579    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:12:35.168433 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.168397    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca2d4b2-04bd-4c4f-be95-223f0718393c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-338a0860-vgdd7\" (UID: \"bca2d4b2-04bd-4c4f-be95-223f0718393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"
Nov 02 13:12:35.168634 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.168585    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca2d4b2-04bd-4c4f-be95-223f0718393c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-338a0860-vgdd7\" (UID: \"bca2d4b2-04bd-4c4f-be95-223f0718393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"
Nov 02 13:12:35.168710 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.168645    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbb75\" (UniqueName: \"kubernetes.io/projected/bca2d4b2-04bd-4c4f-be95-223f0718393c-kube-api-access-cbb75\") pod \"maas-api-revoke-sub-338a0860-vgdd7\" (UID: \"bca2d4b2-04bd-4c4f-be95-223f0718393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"
Nov 02 13:12:35.185449 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.185416    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:12:35.185449 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.185417    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 13:12:35.187991 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.187955    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"]
Nov 02 13:12:35.238395 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.238347    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"]
Nov 02 13:12:35.239822 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.239793    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv" podUID="b7ce9975-1859-40b9-933b-a9f52618008f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:12:35.257485 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.257459    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"]
Nov 02 13:12:35.261426 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.261405    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"
Nov 02 13:12:35.270130 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.270107    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca2d4b2-04bd-4c4f-be95-223f0718393c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-338a0860-vgdd7\" (UID: \"bca2d4b2-04bd-4c4f-be95-223f0718393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"
Nov 02 13:12:35.270244 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.270166    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbb75\" (UniqueName: \"kubernetes.io/projected/bca2d4b2-04bd-4c4f-be95-223f0718393c-kube-api-access-cbb75\") pod \"maas-api-revoke-sub-338a0860-vgdd7\" (UID: \"bca2d4b2-04bd-4c4f-be95-223f0718393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"
Nov 02 13:12:35.270309 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.270241    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca2d4b2-04bd-4c4f-be95-223f0718393c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-338a0860-vgdd7\" (UID: \"bca2d4b2-04bd-4c4f-be95-223f0718393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"
Nov 02 13:12:35.270751 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.270731    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca2d4b2-04bd-4c4f-be95-223f0718393c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-338a0860-vgdd7\" (UID: \"bca2d4b2-04bd-4c4f-be95-223f0718393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"
Nov 02 13:12:35.272508 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.272487    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca2d4b2-04bd-4c4f-be95-223f0718393c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-338a0860-vgdd7\" (UID: \"bca2d4b2-04bd-4c4f-be95-223f0718393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"
Nov 02 13:12:35.318715 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.318681    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"]
Nov 02 13:12:35.336239 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.336174    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbb75\" (UniqueName: \"kubernetes.io/projected/bca2d4b2-04bd-4c4f-be95-223f0718393c-kube-api-access-cbb75\") pod \"maas-api-revoke-sub-338a0860-vgdd7\" (UID: \"bca2d4b2-04bd-4c4f-be95-223f0718393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"
Nov 02 13:12:35.370926 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.370894    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f04ed7-b132-4c5a-b86c-07abafe7364a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4\" (UID: \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"
Nov 02 13:12:35.371108 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.370952    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f04ed7-b132-4c5a-b86c-07abafe7364a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4\" (UID: \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"
Nov 02 13:12:35.371108 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.371074    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f04ed7-b132-4c5a-b86c-07abafe7364a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4\" (UID: \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"
Nov 02 13:12:35.371221 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.371116    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwpnd\" (UniqueName: \"kubernetes.io/projected/a5f04ed7-b132-4c5a-b86c-07abafe7364a-kube-api-access-zwpnd\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4\" (UID: \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"
Nov 02 13:12:35.394780 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.394756    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"
Nov 02 13:12:35.473178 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.472254    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwpnd\" (UniqueName: \"kubernetes.io/projected/a5f04ed7-b132-4c5a-b86c-07abafe7364a-kube-api-access-zwpnd\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4\" (UID: \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"
Nov 02 13:12:35.473178 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.472422    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f04ed7-b132-4c5a-b86c-07abafe7364a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4\" (UID: \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"
Nov 02 13:12:35.473178 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.472465    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f04ed7-b132-4c5a-b86c-07abafe7364a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4\" (UID: \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"
Nov 02 13:12:35.473178 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.472498    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f04ed7-b132-4c5a-b86c-07abafe7364a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4\" (UID: \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"
Nov 02 13:12:35.475241 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.475182    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f04ed7-b132-4c5a-b86c-07abafe7364a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4\" (UID: \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"
Nov 02 13:12:35.475955 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.475929    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f04ed7-b132-4c5a-b86c-07abafe7364a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4\" (UID: \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"
Nov 02 13:12:35.476303 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.476279    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f04ed7-b132-4c5a-b86c-07abafe7364a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4\" (UID: \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"
Nov 02 13:12:35.533808 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.533776    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwpnd\" (UniqueName: \"kubernetes.io/projected/a5f04ed7-b132-4c5a-b86c-07abafe7364a-kube-api-access-zwpnd\") pod \"payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4\" (UID: \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\") " pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"
Nov 02 13:12:35.573898 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.573869    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"
Nov 02 13:12:35.767548 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.767518    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"]
Nov 02 13:12:35.768688 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:12:35.768663    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca2d4b2_04bd_4c4f_be95_223f0718393c.slice/crio-cb5a6afc68ce608b895277bf477577eefb3bd55101ec745d4bf7ae8424fef79c WatchSource:0}: Error finding container cb5a6afc68ce608b895277bf477577eefb3bd55101ec745d4bf7ae8424fef79c: Status 404 returned error can't find the container with id cb5a6afc68ce608b895277bf477577eefb3bd55101ec745d4bf7ae8424fef79c
Nov 02 13:12:35.948031 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:35.948005    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"]
Nov 02 13:12:35.949391 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:12:35.949353    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f04ed7_b132_4c5a_b86c_07abafe7364a.slice/crio-858a66df812fcaa45a96a83d8a3a415383a8ab0bf42fe2f04eab9f1ae6965b1a WatchSource:0}: Error finding container 858a66df812fcaa45a96a83d8a3a415383a8ab0bf42fe2f04eab9f1ae6965b1a: Status 404 returned error can't find the container with id 858a66df812fcaa45a96a83d8a3a415383a8ab0bf42fe2f04eab9f1ae6965b1a
Nov 02 13:12:36.198687 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:36.198588    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4" event={"ID":"a5f04ed7-b132-4c5a-b86c-07abafe7364a","Type":"ContainerStarted","Data":"258c896b0b87d77c257e1bbcb1a9a85b473738ca92eb8df50975041c6434c4f8"}
Nov 02 13:12:36.198687 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:36.198657    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4" event={"ID":"a5f04ed7-b132-4c5a-b86c-07abafe7364a","Type":"ContainerStarted","Data":"858a66df812fcaa45a96a83d8a3a415383a8ab0bf42fe2f04eab9f1ae6965b1a"}
Nov 02 13:12:36.199173 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:36.198798    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"
Nov 02 13:12:36.200238 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:36.200216    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7" event={"ID":"bca2d4b2-04bd-4c4f-be95-223f0718393c","Type":"ContainerStarted","Data":"126bf4e54e1ac432e344604f32670f8f3c17e4ad49ca6e59f1e324f7516c7ece"}
Nov 02 13:12:36.200332 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:36.200252    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7" event={"ID":"bca2d4b2-04bd-4c4f-be95-223f0718393c","Type":"ContainerStarted","Data":"cb5a6afc68ce608b895277bf477577eefb3bd55101ec745d4bf7ae8424fef79c"}
Nov 02 13:12:36.318130 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:36.318078    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4" podStartSLOduration=1.318066344 podStartE2EDuration="1.318066344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:12:36.316463494 +0000 UTC m=+4584.880171904" watchObservedRunningTime="2026-11-02 13:12:36.318066344 +0000 UTC m=+4584.881774754"
Nov 02 13:12:36.434482 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:36.434428    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7" podStartSLOduration=1.434413731 podStartE2EDuration="1.434413731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:12:36.433161406 +0000 UTC m=+4584.996869818" watchObservedRunningTime="2026-11-02 13:12:36.434413731 +0000 UTC m=+4584.998122140"
Nov 02 13:12:37.207339 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:37.207297    4464 generic.go:356] "Generic (PLEG): container finished" podID="bca2d4b2-04bd-4c4f-be95-223f0718393c" containerID="126bf4e54e1ac432e344604f32670f8f3c17e4ad49ca6e59f1e324f7516c7ece" exitCode=0
Nov 02 13:12:37.207804 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:37.207378    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7" event={"ID":"bca2d4b2-04bd-4c4f-be95-223f0718393c","Type":"ContainerDied","Data":"126bf4e54e1ac432e344604f32670f8f3c17e4ad49ca6e59f1e324f7516c7ece"}
Nov 02 13:12:37.823438 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:37.823405    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"]
Nov 02 13:12:37.824587 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:37.824559    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z" podUID="4773433e-d8c6-4bb0-b9e9-00dadae397ca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:12:37.861940 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:37.861913    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"]
Nov 02 13:12:37.865679 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:37.865662    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"
Nov 02 13:12:37.898242 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:37.898212    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lrvv\" (UniqueName: \"kubernetes.io/projected/5726a503-7324-4dd0-b063-2b9d820d63c3-kube-api-access-4lrvv\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b\" (UID: \"5726a503-7324-4dd0-b063-2b9d820d63c3\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"
Nov 02 13:12:37.898348 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:37.898261    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5726a503-7324-4dd0-b063-2b9d820d63c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b\" (UID: \"5726a503-7324-4dd0-b063-2b9d820d63c3\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"
Nov 02 13:12:37.898394 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:37.898369    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5726a503-7324-4dd0-b063-2b9d820d63c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b\" (UID: \"5726a503-7324-4dd0-b063-2b9d820d63c3\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"
Nov 02 13:12:37.898445 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:37.898417    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5726a503-7324-4dd0-b063-2b9d820d63c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b\" (UID: \"5726a503-7324-4dd0-b063-2b9d820d63c3\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"
Nov 02 13:12:37.934052 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:37.934019    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"]
Nov 02 13:12:37.999180 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:37.999147    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrvv\" (UniqueName: \"kubernetes.io/projected/5726a503-7324-4dd0-b063-2b9d820d63c3-kube-api-access-4lrvv\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b\" (UID: \"5726a503-7324-4dd0-b063-2b9d820d63c3\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"
Nov 02 13:12:37.999322 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:37.999190    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5726a503-7324-4dd0-b063-2b9d820d63c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b\" (UID: \"5726a503-7324-4dd0-b063-2b9d820d63c3\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"
Nov 02 13:12:37.999322 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:37.999245    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5726a503-7324-4dd0-b063-2b9d820d63c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b\" (UID: \"5726a503-7324-4dd0-b063-2b9d820d63c3\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"
Nov 02 13:12:37.999322 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:37.999286    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5726a503-7324-4dd0-b063-2b9d820d63c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b\" (UID: \"5726a503-7324-4dd0-b063-2b9d820d63c3\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"
Nov 02 13:12:37.999923 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:37.999900    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5726a503-7324-4dd0-b063-2b9d820d63c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b\" (UID: \"5726a503-7324-4dd0-b063-2b9d820d63c3\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"
Nov 02 13:12:38.000021 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:37.999956    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5726a503-7324-4dd0-b063-2b9d820d63c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b\" (UID: \"5726a503-7324-4dd0-b063-2b9d820d63c3\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"
Nov 02 13:12:38.001714 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:38.001696    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5726a503-7324-4dd0-b063-2b9d820d63c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b\" (UID: \"5726a503-7324-4dd0-b063-2b9d820d63c3\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"
Nov 02 13:12:38.061296 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:38.061265    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lrvv\" (UniqueName: \"kubernetes.io/projected/5726a503-7324-4dd0-b063-2b9d820d63c3-kube-api-access-4lrvv\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b\" (UID: \"5726a503-7324-4dd0-b063-2b9d820d63c3\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"
Nov 02 13:12:38.178909 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:38.178835    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"
Nov 02 13:12:38.364408 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:38.364384    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"
Nov 02 13:12:38.404413 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:38.404387    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bca2d4b2-04bd-4c4f-be95-223f0718393c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca2d4b2-04bd-4c4f-be95-223f0718393c-maas-api-xks-ca\") pod \"bca2d4b2-04bd-4c4f-be95-223f0718393c\" (UID: \"bca2d4b2-04bd-4c4f-be95-223f0718393c\") "
Nov 02 13:12:38.404535 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:38.404471    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bca2d4b2-04bd-4c4f-be95-223f0718393c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca2d4b2-04bd-4c4f-be95-223f0718393c-maas-api-openshift-ca\") pod \"bca2d4b2-04bd-4c4f-be95-223f0718393c\" (UID: \"bca2d4b2-04bd-4c4f-be95-223f0718393c\") "
Nov 02 13:12:38.404580 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:38.404531    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bca2d4b2-04bd-4c4f-be95-223f0718393c-kube-api-access-cbb75\" (UniqueName: \"kubernetes.io/projected/bca2d4b2-04bd-4c4f-be95-223f0718393c-kube-api-access-cbb75\") pod \"bca2d4b2-04bd-4c4f-be95-223f0718393c\" (UID: \"bca2d4b2-04bd-4c4f-be95-223f0718393c\") "
Nov 02 13:12:38.404821 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:38.404792    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca2d4b2-04bd-4c4f-be95-223f0718393c-maas-api-openshift-ca" pod "bca2d4b2-04bd-4c4f-be95-223f0718393c" (UID: "bca2d4b2-04bd-4c4f-be95-223f0718393c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:12:38.406362 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:38.406341    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca2d4b2-04bd-4c4f-be95-223f0718393c-maas-api-xks-ca" pod "bca2d4b2-04bd-4c4f-be95-223f0718393c" (UID: "bca2d4b2-04bd-4c4f-be95-223f0718393c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:12:38.406444 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:38.406431    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca2d4b2-04bd-4c4f-be95-223f0718393c-kube-api-access-cbb75" pod "bca2d4b2-04bd-4c4f-be95-223f0718393c" (UID: "bca2d4b2-04bd-4c4f-be95-223f0718393c"). InnerVolumeSpecName "kube-api-access-cbb75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:12:38.505657 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:38.505626    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca2d4b2-04bd-4c4f-be95-223f0718393c-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:38.505657 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:38.505651    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca2d4b2-04bd-4c4f-be95-223f0718393c-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:38.505657 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:38.505662    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbb75\" (UniqueName: \"kubernetes.io/projected/bca2d4b2-04bd-4c4f-be95-223f0718393c-kube-api-access-cbb75\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:38.596435 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:38.596405    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"]
Nov 02 13:12:39.220295 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:39.220261    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b" event={"ID":"5726a503-7324-4dd0-b063-2b9d820d63c3","Type":"ContainerStarted","Data":"c63c7f03b1afe9c00cd9acd8e96db8a96f666ca23e85eacb0772ea378101effc"}
Nov 02 13:12:39.220517 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:39.220324    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b" event={"ID":"5726a503-7324-4dd0-b063-2b9d820d63c3","Type":"ContainerStarted","Data":"05c447b46b772bb589c3b41b10355aba7454e4ff77ae9d3deb16eca64dc3534d"}
Nov 02 13:12:39.220517 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:39.220442    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"
Nov 02 13:12:39.221634 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:39.221614    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"
Nov 02 13:12:39.221753 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:39.221662    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7" event={"ID":"bca2d4b2-04bd-4c4f-be95-223f0718393c","Type":"ContainerDied","Data":"cb5a6afc68ce608b895277bf477577eefb3bd55101ec745d4bf7ae8424fef79c"}
Nov 02 13:12:39.221753 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:39.221700    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb5a6afc68ce608b895277bf477577eefb3bd55101ec745d4bf7ae8424fef79c"
Nov 02 13:12:39.340554 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:39.340453    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b" podStartSLOduration=2.340440471 podStartE2EDuration="2.340440471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:12:39.338869685 +0000 UTC m=+4587.902578094" watchObservedRunningTime="2026-11-02 13:12:39.340440471 +0000 UTC m=+4587.904148879"
Nov 02 13:12:40.239009 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:40.238954    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv" podUID="b7ce9975-1859-40b9-933b-a9f52618008f" containerName="payload-processing" containerID="cri-o://6c02b2a7b749d6cc6d00f58f4920431e21d4d91e734762ee6bec44b21c0748c2" gracePeriod=25
Nov 02 13:12:40.479848 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:40.479822    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"
Nov 02 13:12:40.527454 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:40.527390    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7ce9975-1859-40b9-933b-a9f52618008f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7ce9975-1859-40b9-933b-a9f52618008f-plugins-config-volume\") pod \"b7ce9975-1859-40b9-933b-a9f52618008f\" (UID: \"b7ce9975-1859-40b9-933b-a9f52618008f\") "
Nov 02 13:12:40.527588 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:40.527452    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ce9975-1859-40b9-933b-a9f52618008f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7ce9975-1859-40b9-933b-a9f52618008f-provider-credentials\") pod \"b7ce9975-1859-40b9-933b-a9f52618008f\" (UID: \"b7ce9975-1859-40b9-933b-a9f52618008f\") "
Nov 02 13:12:40.527588 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:40.527516    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7ce9975-1859-40b9-933b-a9f52618008f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7ce9975-1859-40b9-933b-a9f52618008f-routing-overlay-volume\") pod \"b7ce9975-1859-40b9-933b-a9f52618008f\" (UID: \"b7ce9975-1859-40b9-933b-a9f52618008f\") "
Nov 02 13:12:40.527588 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:40.527570    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ce9975-1859-40b9-933b-a9f52618008f-kube-api-access-87btw\" (UniqueName: \"kubernetes.io/projected/b7ce9975-1859-40b9-933b-a9f52618008f-kube-api-access-87btw\") pod \"b7ce9975-1859-40b9-933b-a9f52618008f\" (UID: \"b7ce9975-1859-40b9-933b-a9f52618008f\") "
Nov 02 13:12:40.527774 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:40.527744    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7ce9975-1859-40b9-933b-a9f52618008f-plugins-config-volume" pod "b7ce9975-1859-40b9-933b-a9f52618008f" (UID: "b7ce9975-1859-40b9-933b-a9f52618008f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:12:40.527928 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:40.527907    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7ce9975-1859-40b9-933b-a9f52618008f-routing-overlay-volume" pod "b7ce9975-1859-40b9-933b-a9f52618008f" (UID: "b7ce9975-1859-40b9-933b-a9f52618008f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:12:40.528011 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:40.527930    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7ce9975-1859-40b9-933b-a9f52618008f-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:40.529562 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:40.529543    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ce9975-1859-40b9-933b-a9f52618008f-provider-credentials" pod "b7ce9975-1859-40b9-933b-a9f52618008f" (UID: "b7ce9975-1859-40b9-933b-a9f52618008f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:12:40.529649 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:40.529632    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ce9975-1859-40b9-933b-a9f52618008f-kube-api-access-87btw" pod "b7ce9975-1859-40b9-933b-a9f52618008f" (UID: "b7ce9975-1859-40b9-933b-a9f52618008f"). InnerVolumeSpecName "kube-api-access-87btw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:12:40.628832 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:40.628801    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87btw\" (UniqueName: \"kubernetes.io/projected/b7ce9975-1859-40b9-933b-a9f52618008f-kube-api-access-87btw\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:40.628832 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:40.628827    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b7ce9975-1859-40b9-933b-a9f52618008f-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:40.628832 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:40.628839    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b7ce9975-1859-40b9-933b-a9f52618008f-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:41.233313 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:41.233280    4464 generic.go:356] "Generic (PLEG): container finished" podID="b7ce9975-1859-40b9-933b-a9f52618008f" containerID="6c02b2a7b749d6cc6d00f58f4920431e21d4d91e734762ee6bec44b21c0748c2" exitCode=0
Nov 02 13:12:41.233536 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:41.233333    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"
Nov 02 13:12:41.233536 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:41.233340    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv" event={"ID":"b7ce9975-1859-40b9-933b-a9f52618008f","Type":"ContainerDied","Data":"6c02b2a7b749d6cc6d00f58f4920431e21d4d91e734762ee6bec44b21c0748c2"}
Nov 02 13:12:41.233536 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:41.233392    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv" event={"ID":"b7ce9975-1859-40b9-933b-a9f52618008f","Type":"ContainerDied","Data":"ec9f40a0d057c6b85272428ed174cc8ad5c2188f2cf5aab38165e58504ae0fca"}
Nov 02 13:12:41.233536 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:41.233421    4464 scope.go:122] "RemoveContainer" containerID="6c02b2a7b749d6cc6d00f58f4920431e21d4d91e734762ee6bec44b21c0748c2"
Nov 02 13:12:41.243863 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:41.243689    4464 scope.go:122] "RemoveContainer" containerID="6c02b2a7b749d6cc6d00f58f4920431e21d4d91e734762ee6bec44b21c0748c2"
Nov 02 13:12:41.244135 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:41.243963    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c02b2a7b749d6cc6d00f58f4920431e21d4d91e734762ee6bec44b21c0748c2\": container with ID starting with 6c02b2a7b749d6cc6d00f58f4920431e21d4d91e734762ee6bec44b21c0748c2 not found: ID does not exist" containerID="6c02b2a7b749d6cc6d00f58f4920431e21d4d91e734762ee6bec44b21c0748c2"
Nov 02 13:12:41.244135 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:41.244015    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c02b2a7b749d6cc6d00f58f4920431e21d4d91e734762ee6bec44b21c0748c2"} err="failed to get container status \"6c02b2a7b749d6cc6d00f58f4920431e21d4d91e734762ee6bec44b21c0748c2\": rpc error: code = NotFound desc = could not find container \"6c02b2a7b749d6cc6d00f58f4920431e21d4d91e734762ee6bec44b21c0748c2\": container with ID starting with 6c02b2a7b749d6cc6d00f58f4920431e21d4d91e734762ee6bec44b21c0748c2 not found: ID does not exist"
Nov 02 13:12:41.401305 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:41.401276    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"]
Nov 02 13:12:41.451815 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:41.451782    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-7c78schqv"]
Nov 02 13:12:41.588116 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:41.588039    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7ce9975-1859-40b9-933b-a9f52618008f" path="/var/lib/kubelet/pods/b7ce9975-1859-40b9-933b-a9f52618008f/volumes"
Nov 02 13:12:42.823788 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:42.823746    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z" podUID="4773433e-d8c6-4bb0-b9e9-00dadae397ca" containerName="payload-processing" containerID="cri-o://ebeea8615322436d12c5dfaa48a7b6876c9c564a0ebd2e47c682a46f75096ca3" gracePeriod=25
Nov 02 13:12:43.074270 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.074219    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"
Nov 02 13:12:43.150824 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.150797    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4773433e-d8c6-4bb0-b9e9-00dadae397ca-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4773433e-d8c6-4bb0-b9e9-00dadae397ca-routing-overlay-volume\") pod \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\" (UID: \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\") "
Nov 02 13:12:43.150961 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.150909    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4773433e-d8c6-4bb0-b9e9-00dadae397ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4773433e-d8c6-4bb0-b9e9-00dadae397ca-plugins-config-volume\") pod \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\" (UID: \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\") "
Nov 02 13:12:43.151040 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.150967    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4773433e-d8c6-4bb0-b9e9-00dadae397ca-kube-api-access-xqhrg\" (UniqueName: \"kubernetes.io/projected/4773433e-d8c6-4bb0-b9e9-00dadae397ca-kube-api-access-xqhrg\") pod \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\" (UID: \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\") "
Nov 02 13:12:43.151105 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.151038    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4773433e-d8c6-4bb0-b9e9-00dadae397ca-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4773433e-d8c6-4bb0-b9e9-00dadae397ca-provider-credentials\") pod \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\" (UID: \"4773433e-d8c6-4bb0-b9e9-00dadae397ca\") "
Nov 02 13:12:43.151238 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.151215    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4773433e-d8c6-4bb0-b9e9-00dadae397ca-routing-overlay-volume" pod "4773433e-d8c6-4bb0-b9e9-00dadae397ca" (UID: "4773433e-d8c6-4bb0-b9e9-00dadae397ca"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:12:43.151310 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.151290    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4773433e-d8c6-4bb0-b9e9-00dadae397ca-plugins-config-volume" pod "4773433e-d8c6-4bb0-b9e9-00dadae397ca" (UID: "4773433e-d8c6-4bb0-b9e9-00dadae397ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:12:43.151413 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.151398    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4773433e-d8c6-4bb0-b9e9-00dadae397ca-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:43.151449 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.151416    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4773433e-d8c6-4bb0-b9e9-00dadae397ca-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:43.153129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.153106    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4773433e-d8c6-4bb0-b9e9-00dadae397ca-provider-credentials" pod "4773433e-d8c6-4bb0-b9e9-00dadae397ca" (UID: "4773433e-d8c6-4bb0-b9e9-00dadae397ca"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:12:43.153241 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.153222    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4773433e-d8c6-4bb0-b9e9-00dadae397ca-kube-api-access-xqhrg" pod "4773433e-d8c6-4bb0-b9e9-00dadae397ca" (UID: "4773433e-d8c6-4bb0-b9e9-00dadae397ca"). InnerVolumeSpecName "kube-api-access-xqhrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:12:43.246873 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.246837    4464 generic.go:356] "Generic (PLEG): container finished" podID="4773433e-d8c6-4bb0-b9e9-00dadae397ca" containerID="ebeea8615322436d12c5dfaa48a7b6876c9c564a0ebd2e47c682a46f75096ca3" exitCode=0
Nov 02 13:12:43.247060 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.246912    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"
Nov 02 13:12:43.247060 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.246913    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z" event={"ID":"4773433e-d8c6-4bb0-b9e9-00dadae397ca","Type":"ContainerDied","Data":"ebeea8615322436d12c5dfaa48a7b6876c9c564a0ebd2e47c682a46f75096ca3"}
Nov 02 13:12:43.247060 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.247031    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z" event={"ID":"4773433e-d8c6-4bb0-b9e9-00dadae397ca","Type":"ContainerDied","Data":"727934eeb7fa7d9ae06beca9be77ca153e2562207f4e7799de715c35d8034745"}
Nov 02 13:12:43.247060 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.247053    4464 scope.go:122] "RemoveContainer" containerID="ebeea8615322436d12c5dfaa48a7b6876c9c564a0ebd2e47c682a46f75096ca3"
Nov 02 13:12:43.251938 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.251920    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqhrg\" (UniqueName: \"kubernetes.io/projected/4773433e-d8c6-4bb0-b9e9-00dadae397ca-kube-api-access-xqhrg\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:43.251938 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.251938    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4773433e-d8c6-4bb0-b9e9-00dadae397ca-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:43.257413 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.257397    4464 scope.go:122] "RemoveContainer" containerID="ebeea8615322436d12c5dfaa48a7b6876c9c564a0ebd2e47c682a46f75096ca3"
Nov 02 13:12:43.257741 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:43.257721    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebeea8615322436d12c5dfaa48a7b6876c9c564a0ebd2e47c682a46f75096ca3\": container with ID starting with ebeea8615322436d12c5dfaa48a7b6876c9c564a0ebd2e47c682a46f75096ca3 not found: ID does not exist" containerID="ebeea8615322436d12c5dfaa48a7b6876c9c564a0ebd2e47c682a46f75096ca3"
Nov 02 13:12:43.257822 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.257746    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebeea8615322436d12c5dfaa48a7b6876c9c564a0ebd2e47c682a46f75096ca3"} err="failed to get container status \"ebeea8615322436d12c5dfaa48a7b6876c9c564a0ebd2e47c682a46f75096ca3\": rpc error: code = NotFound desc = could not find container \"ebeea8615322436d12c5dfaa48a7b6876c9c564a0ebd2e47c682a46f75096ca3\": container with ID starting with ebeea8615322436d12c5dfaa48a7b6876c9c564a0ebd2e47c682a46f75096ca3 not found: ID does not exist"
Nov 02 13:12:43.416253 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.416172    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"]
Nov 02 13:12:43.422570 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.422539    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-589csdp6z"]
Nov 02 13:12:43.581036 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.581002    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4773433e-d8c6-4bb0-b9e9-00dadae397ca" path="/var/lib/kubelet/pods/4773433e-d8c6-4bb0-b9e9-00dadae397ca/volumes"
Nov 02 13:12:43.876753 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:43.876716    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:12:46.770224 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.770191    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"]
Nov 02 13:12:46.770672 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.770658    4464 cpu_manager.go:414] "containerMap: removing container" podUID="4773433e-d8c6-4bb0-b9e9-00dadae397ca" containerName="payload-processing"
Nov 02 13:12:46.770726 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.770673    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4773433e-d8c6-4bb0-b9e9-00dadae397ca" containerName="payload-processing"
Nov 02 13:12:46.770726 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.770687    4464 cpu_manager.go:414] "containerMap: removing container" podUID="b7ce9975-1859-40b9-933b-a9f52618008f" containerName="payload-processing"
Nov 02 13:12:46.770726 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.770692    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7ce9975-1859-40b9-933b-a9f52618008f" containerName="payload-processing"
Nov 02 13:12:46.770726 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.770725    4464 cpu_manager.go:414] "containerMap: removing container" podUID="bca2d4b2-04bd-4c4f-be95-223f0718393c" containerName="revoke-keys"
Nov 02 13:12:46.770870 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.770731    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca2d4b2-04bd-4c4f-be95-223f0718393c" containerName="revoke-keys"
Nov 02 13:12:46.770870 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.770794    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca2d4b2-04bd-4c4f-be95-223f0718393c" containerName="revoke-keys"
Nov 02 13:12:46.770870 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.770802    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="4773433e-d8c6-4bb0-b9e9-00dadae397ca" containerName="payload-processing"
Nov 02 13:12:46.770870 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.770810    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7ce9975-1859-40b9-933b-a9f52618008f" containerName="payload-processing"
Nov 02 13:12:46.775288 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.775272    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"
Nov 02 13:12:46.828487 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.828461    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:12:46.879325 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.879294    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:12:46.879325 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.879301    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\""
Nov 02 13:12:46.879511 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.879353    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:12:46.881824 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.881802    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"]
Nov 02 13:12:46.886846 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.886804    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j8dl\" (UniqueName: \"kubernetes.io/projected/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-kube-api-access-8j8dl\") pod \"maas-api-revoke-sub-a3ca2812-9pnlj\" (UID: \"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"
Nov 02 13:12:46.886946 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.886910    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3ca2812-9pnlj\" (UID: \"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"
Nov 02 13:12:46.887015 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.886955    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3ca2812-9pnlj\" (UID: \"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"
Nov 02 13:12:46.988001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.987948    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j8dl\" (UniqueName: \"kubernetes.io/projected/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-kube-api-access-8j8dl\") pod \"maas-api-revoke-sub-a3ca2812-9pnlj\" (UID: \"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"
Nov 02 13:12:46.988173 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.988040    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3ca2812-9pnlj\" (UID: \"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"
Nov 02 13:12:46.988173 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.988091    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3ca2812-9pnlj\" (UID: \"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"
Nov 02 13:12:46.988689 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.988661    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3ca2812-9pnlj\" (UID: \"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"
Nov 02 13:12:46.990372 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:46.990351    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3ca2812-9pnlj\" (UID: \"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"
Nov 02 13:12:47.050036 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:47.049948    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j8dl\" (UniqueName: \"kubernetes.io/projected/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-kube-api-access-8j8dl\") pod \"maas-api-revoke-sub-a3ca2812-9pnlj\" (UID: \"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"
Nov 02 13:12:47.087981 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:47.087957    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"
Nov 02 13:12:47.210727 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:47.210686    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4" podUID="a5f04ed7-b132-4c5a-b86c-07abafe7364a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:12:47.460053 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:47.460027    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"]
Nov 02 13:12:47.460854 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:12:47.460826    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4620cb6f_7c27_4e6a_b2a2_caea0b4976f0.slice/crio-42cdfcc98bf48dd1efdc46e83c1f9b2af26c04852da7910070da6161c7fc474e WatchSource:0}: Error finding container 42cdfcc98bf48dd1efdc46e83c1f9b2af26c04852da7910070da6161c7fc474e: Status 404 returned error can't find the container with id 42cdfcc98bf48dd1efdc46e83c1f9b2af26c04852da7910070da6161c7fc474e
Nov 02 13:12:48.277463 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:48.277425    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj" event={"ID":"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0","Type":"ContainerStarted","Data":"0d76458c3954a9916bad9a82f317e8418b86da057daa972bb6f025546f2de8e2"}
Nov 02 13:12:48.277853 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:48.277479    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj" event={"ID":"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0","Type":"ContainerStarted","Data":"42cdfcc98bf48dd1efdc46e83c1f9b2af26c04852da7910070da6161c7fc474e"}
Nov 02 13:12:48.396043 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:48.395994    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj" podStartSLOduration=2.395962078 podStartE2EDuration="2.395962078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:12:48.394875627 +0000 UTC m=+4596.958584038" watchObservedRunningTime="2026-11-02 13:12:48.395962078 +0000 UTC m=+4596.959670486"
Nov 02 13:12:49.284105 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:49.284073    4464 generic.go:356] "Generic (PLEG): container finished" podID="4620cb6f-7c27-4e6a-b2a2-caea0b4976f0" containerID="0d76458c3954a9916bad9a82f317e8418b86da057daa972bb6f025546f2de8e2" exitCode=0
Nov 02 13:12:49.284548 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:49.284161    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj" event={"ID":"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0","Type":"ContainerDied","Data":"0d76458c3954a9916bad9a82f317e8418b86da057daa972bb6f025546f2de8e2"}
Nov 02 13:12:50.229511 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:50.229473    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b" podUID="5726a503-7324-4dd0-b063-2b9d820d63c3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:12:50.419613 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:50.419585    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"
Nov 02 13:12:50.519810 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:50.519741    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-maas-api-xks-ca\") pod \"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0\" (UID: \"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0\") "
Nov 02 13:12:50.519969 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:50.519849    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-kube-api-access-8j8dl\" (UniqueName: \"kubernetes.io/projected/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-kube-api-access-8j8dl\") pod \"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0\" (UID: \"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0\") "
Nov 02 13:12:50.519969 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:50.519875    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-maas-api-openshift-ca\") pod \"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0\" (UID: \"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0\") "
Nov 02 13:12:50.520304 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:50.520280    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-maas-api-openshift-ca" pod "4620cb6f-7c27-4e6a-b2a2-caea0b4976f0" (UID: "4620cb6f-7c27-4e6a-b2a2-caea0b4976f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:12:50.521793 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:50.521768    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-maas-api-xks-ca" pod "4620cb6f-7c27-4e6a-b2a2-caea0b4976f0" (UID: "4620cb6f-7c27-4e6a-b2a2-caea0b4976f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:12:50.521953 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:50.521931    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-kube-api-access-8j8dl" pod "4620cb6f-7c27-4e6a-b2a2-caea0b4976f0" (UID: "4620cb6f-7c27-4e6a-b2a2-caea0b4976f0"). InnerVolumeSpecName "kube-api-access-8j8dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:12:50.621698 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:50.621664    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:50.621698 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:50.621695    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j8dl\" (UniqueName: \"kubernetes.io/projected/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-kube-api-access-8j8dl\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:50.621698 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:50.621705    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:12:51.296883 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:51.296850    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"
Nov 02 13:12:51.297073 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:51.296879    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj" event={"ID":"4620cb6f-7c27-4e6a-b2a2-caea0b4976f0","Type":"ContainerDied","Data":"42cdfcc98bf48dd1efdc46e83c1f9b2af26c04852da7910070da6161c7fc474e"}
Nov 02 13:12:51.297073 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:51.296929    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42cdfcc98bf48dd1efdc46e83c1f9b2af26c04852da7910070da6161c7fc474e"
Nov 02 13:12:56.682929 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:56.682891    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-2a587a2bd360: secret "agc-cred-2a587a2bd360" not found
Nov 02 13:12:56.682929 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:56.682927    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-32cb2b2dfd51: secret "agc-cred-32cb2b2dfd51" not found
Nov 02 13:12:56.682929 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:56.682937    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-add3ebb37edc: secret "agc-cred-add3ebb37edc" not found
Nov 02 13:12:56.683554 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:56.682951    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w: [secret "agc-cred-2a587a2bd360" not found, secret "agc-cred-32cb2b2dfd51" not found, secret "agc-cred-add3ebb37edc" not found]
Nov 02 13:12:56.683554 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:56.683047    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-provider-credentials podName:7b4696e4-c36d-4e16-bb64-cb1697e320ce nodeName:}" failed. No retries permitted until 2026-11-02 13:12:57.183023307 +0000 UTC m=+4605.746731697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w" (UID: "7b4696e4-c36d-4e16-bb64-cb1697e320ce") : [secret "agc-cred-2a587a2bd360" not found, secret "agc-cred-32cb2b2dfd51" not found, secret "agc-cred-add3ebb37edc" not found]
Nov 02 13:12:57.007006 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:57.006962    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:12:57.188703 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:57.188665    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-2a587a2bd360: secret "agc-cred-2a587a2bd360" not found
Nov 02 13:12:57.188703 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:57.188694    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-32cb2b2dfd51: secret "agc-cred-32cb2b2dfd51" not found
Nov 02 13:12:57.188703 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:57.188706    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-add3ebb37edc: secret "agc-cred-add3ebb37edc" not found
Nov 02 13:12:57.189022 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:57.188719    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w: [secret "agc-cred-2a587a2bd360" not found, secret "agc-cred-32cb2b2dfd51" not found, secret "agc-cred-add3ebb37edc" not found]
Nov 02 13:12:57.189022 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:57.188785    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-provider-credentials podName:7b4696e4-c36d-4e16-bb64-cb1697e320ce nodeName:}" failed. No retries permitted until 2026-11-02 13:12:58.188765029 +0000 UTC m=+4606.752473417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w" (UID: "7b4696e4-c36d-4e16-bb64-cb1697e320ce") : [secret "agc-cred-2a587a2bd360" not found, secret "agc-cred-32cb2b2dfd51" not found, secret "agc-cred-add3ebb37edc" not found]
Nov 02 13:12:57.209471 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:57.209436    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4" podUID="a5f04ed7-b132-4c5a-b86c-07abafe7364a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:12:58.199442 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:58.199412    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-2a587a2bd360: secret "agc-cred-2a587a2bd360" not found
Nov 02 13:12:58.199442 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:58.199437    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-32cb2b2dfd51: secret "agc-cred-32cb2b2dfd51" not found
Nov 02 13:12:58.199442 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:58.199444    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-add3ebb37edc: secret "agc-cred-add3ebb37edc" not found
Nov 02 13:12:58.200006 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:58.199454    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w: [secret "agc-cred-2a587a2bd360" not found, secret "agc-cred-32cb2b2dfd51" not found, secret "agc-cred-add3ebb37edc" not found]
Nov 02 13:12:58.200006 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:58.199511    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-provider-credentials podName:7b4696e4-c36d-4e16-bb64-cb1697e320ce nodeName:}" failed. No retries permitted until 2026-11-02 13:13:00.199496393 +0000 UTC m=+4608.763204779 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w" (UID: "7b4696e4-c36d-4e16-bb64-cb1697e320ce") : [secret "agc-cred-2a587a2bd360" not found, secret "agc-cred-32cb2b2dfd51" not found, secret "agc-cred-add3ebb37edc" not found]
Nov 02 13:12:58.292026 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:12:58.291989    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:12:59.583518 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:59.583486    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"]
Nov 02 13:12:59.585811 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:59.584721    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4" podUID="a5f04ed7-b132-4c5a-b86c-07abafe7364a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:12:59.596878 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:12:59.596856    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"]
Nov 02 13:13:00.219460 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:00.219431    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-0994b6920024: secret "agc-cred-0994b6920024" not found
Nov 02 13:13:00.219460 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:00.219459    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-2a587a2bd360: secret "agc-cred-2a587a2bd360" not found
Nov 02 13:13:00.219460 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:00.219467    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-32cb2b2dfd51: secret "agc-cred-32cb2b2dfd51" not found
Nov 02 13:13:00.219713 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:00.219474    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-add3ebb37edc: secret "agc-cred-add3ebb37edc" not found
Nov 02 13:13:00.219713 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:00.219482    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w: [secret "agc-cred-0994b6920024" not found, secret "agc-cred-2a587a2bd360" not found, secret "agc-cred-32cb2b2dfd51" not found, secret "agc-cred-add3ebb37edc" not found]
Nov 02 13:13:00.219713 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:00.219540    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-provider-credentials podName:7b4696e4-c36d-4e16-bb64-cb1697e320ce nodeName:}" failed. No retries permitted until 2026-11-02 13:13:04.2195257 +0000 UTC m=+4612.783234113 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w" (UID: "7b4696e4-c36d-4e16-bb64-cb1697e320ce") : [secret "agc-cred-0994b6920024" not found, secret "agc-cred-2a587a2bd360" not found, secret "agc-cred-32cb2b2dfd51" not found, secret "agc-cred-add3ebb37edc" not found]
Nov 02 13:13:00.228068 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:00.228040    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b" podUID="5726a503-7324-4dd0-b063-2b9d820d63c3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:13:02.352600 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:02.352566    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"]
Nov 02 13:13:02.354130 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:02.354099    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b" podUID="5726a503-7324-4dd0-b063-2b9d820d63c3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:13:02.403309 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:02.403281    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"]
Nov 02 13:13:04.264792 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:04.264757    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-0994b6920024: secret "agc-cred-0994b6920024" not found
Nov 02 13:13:04.264792 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:04.264785    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-2a587a2bd360: secret "agc-cred-2a587a2bd360" not found
Nov 02 13:13:04.264792 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:04.264792    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-32cb2b2dfd51: secret "agc-cred-32cb2b2dfd51" not found
Nov 02 13:13:04.264792 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:04.264798    4464 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6ac03f/agc-cred-add3ebb37edc: secret "agc-cred-add3ebb37edc" not found
Nov 02 13:13:04.265327 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:04.264808    4464 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w: [secret "agc-cred-0994b6920024" not found, secret "agc-cred-2a587a2bd360" not found, secret "agc-cred-32cb2b2dfd51" not found, secret "agc-cred-add3ebb37edc" not found]
Nov 02 13:13:04.265327 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:04.264866    4464 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-provider-credentials podName:7b4696e4-c36d-4e16-bb64-cb1697e320ce nodeName:}" failed. No retries permitted until 2026-11-02 13:13:12.264851737 +0000 UTC m=+4620.828560137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w" (UID: "7b4696e4-c36d-4e16-bb64-cb1697e320ce") : [secret "agc-cred-0994b6920024" not found, secret "agc-cred-2a587a2bd360" not found, secret "agc-cred-32cb2b2dfd51" not found, secret "agc-cred-add3ebb37edc" not found]
Nov 02 13:13:04.584122 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.584039    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4" podUID="a5f04ed7-b132-4c5a-b86c-07abafe7364a" containerName="payload-processing" containerID="cri-o://258c896b0b87d77c257e1bbcb1a9a85b473738ca92eb8df50975041c6434c4f8" gracePeriod=25
Nov 02 13:13:04.597307 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.597272    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w" podUID="7b4696e4-c36d-4e16-bb64-cb1697e320ce" containerName="payload-processing" containerID="cri-o://000b755a77737abf603701afea6768f6b80c36c01ea607333d61a81dc50f8198" gracePeriod=25
Nov 02 13:13:04.861235 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.861212    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"
Nov 02 13:13:04.864360 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.864338    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:13:04.970570 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.970539    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f04ed7-b132-4c5a-b86c-07abafe7364a-kube-api-access-zwpnd\" (UniqueName: \"kubernetes.io/projected/a5f04ed7-b132-4c5a-b86c-07abafe7364a-kube-api-access-zwpnd\") pod \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\" (UID: \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\") "
Nov 02 13:13:04.970787 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.970592    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-kube-api-access-x2hbx\" (UniqueName: \"kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-kube-api-access-x2hbx\") pod \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\" (UID: \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\") "
Nov 02 13:13:04.970787 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.970717    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f04ed7-b132-4c5a-b86c-07abafe7364a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f04ed7-b132-4c5a-b86c-07abafe7364a-provider-credentials\") pod \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\" (UID: \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\") "
Nov 02 13:13:04.970930 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.970781    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f04ed7-b132-4c5a-b86c-07abafe7364a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f04ed7-b132-4c5a-b86c-07abafe7364a-routing-overlay-volume\") pod \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\" (UID: \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\") "
Nov 02 13:13:04.970930 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.970901    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f04ed7-b132-4c5a-b86c-07abafe7364a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f04ed7-b132-4c5a-b86c-07abafe7364a-plugins-config-volume\") pod \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\" (UID: \"a5f04ed7-b132-4c5a-b86c-07abafe7364a\") "
Nov 02 13:13:04.971087 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.970946    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4696e4-c36d-4e16-bb64-cb1697e320ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4696e4-c36d-4e16-bb64-cb1697e320ce-plugins-config-volume\") pod \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\" (UID: \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\") "
Nov 02 13:13:04.971087 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.971015    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-provider-credentials\") pod \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\" (UID: \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\") "
Nov 02 13:13:04.971087 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.971055    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4696e4-c36d-4e16-bb64-cb1697e320ce-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b4696e4-c36d-4e16-bb64-cb1697e320ce-routing-overlay-volume\") pod \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\" (UID: \"7b4696e4-c36d-4e16-bb64-cb1697e320ce\") "
Nov 02 13:13:04.971336 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.971311    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f04ed7-b132-4c5a-b86c-07abafe7364a-routing-overlay-volume" pod "a5f04ed7-b132-4c5a-b86c-07abafe7364a" (UID: "a5f04ed7-b132-4c5a-b86c-07abafe7364a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:04.971736 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.971424    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4696e4-c36d-4e16-bb64-cb1697e320ce-plugins-config-volume" pod "7b4696e4-c36d-4e16-bb64-cb1697e320ce" (UID: "7b4696e4-c36d-4e16-bb64-cb1697e320ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:04.971736 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.971470    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f04ed7-b132-4c5a-b86c-07abafe7364a-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:04.971736 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.971662    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4696e4-c36d-4e16-bb64-cb1697e320ce-routing-overlay-volume" pod "7b4696e4-c36d-4e16-bb64-cb1697e320ce" (UID: "7b4696e4-c36d-4e16-bb64-cb1697e320ce"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:04.971736 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.971682    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f04ed7-b132-4c5a-b86c-07abafe7364a-plugins-config-volume" pod "a5f04ed7-b132-4c5a-b86c-07abafe7364a" (UID: "a5f04ed7-b132-4c5a-b86c-07abafe7364a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:04.973056 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.973035    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-kube-api-access-x2hbx" pod "7b4696e4-c36d-4e16-bb64-cb1697e320ce" (UID: "7b4696e4-c36d-4e16-bb64-cb1697e320ce"). InnerVolumeSpecName "kube-api-access-x2hbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:04.973150 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.973055    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f04ed7-b132-4c5a-b86c-07abafe7364a-kube-api-access-zwpnd" pod "a5f04ed7-b132-4c5a-b86c-07abafe7364a" (UID: "a5f04ed7-b132-4c5a-b86c-07abafe7364a"). InnerVolumeSpecName "kube-api-access-zwpnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:04.973663 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.973649    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-provider-credentials" pod "7b4696e4-c36d-4e16-bb64-cb1697e320ce" (UID: "7b4696e4-c36d-4e16-bb64-cb1697e320ce"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:04.973759 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:04.973737    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f04ed7-b132-4c5a-b86c-07abafe7364a-provider-credentials" pod "a5f04ed7-b132-4c5a-b86c-07abafe7364a" (UID: "a5f04ed7-b132-4c5a-b86c-07abafe7364a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:05.072309 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.072271    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f04ed7-b132-4c5a-b86c-07abafe7364a-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:05.072309 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.072302    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f04ed7-b132-4c5a-b86c-07abafe7364a-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:05.072522 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.072319    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4696e4-c36d-4e16-bb64-cb1697e320ce-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:05.072522 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.072331    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:05.072522 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.072346    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b4696e4-c36d-4e16-bb64-cb1697e320ce-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:05.072522 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.072358    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwpnd\" (UniqueName: \"kubernetes.io/projected/a5f04ed7-b132-4c5a-b86c-07abafe7364a-kube-api-access-zwpnd\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:05.072522 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.072370    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2hbx\" (UniqueName: \"kubernetes.io/projected/7b4696e4-c36d-4e16-bb64-cb1697e320ce-kube-api-access-x2hbx\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:05.374178 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.374135    4464 generic.go:356] "Generic (PLEG): container finished" podID="7b4696e4-c36d-4e16-bb64-cb1697e320ce" containerID="000b755a77737abf603701afea6768f6b80c36c01ea607333d61a81dc50f8198" exitCode=0
Nov 02 13:13:05.374684 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.374209    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"
Nov 02 13:13:05.374684 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.374221    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w" event={"ID":"7b4696e4-c36d-4e16-bb64-cb1697e320ce","Type":"ContainerDied","Data":"000b755a77737abf603701afea6768f6b80c36c01ea607333d61a81dc50f8198"}
Nov 02 13:13:05.374684 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.374307    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w" event={"ID":"7b4696e4-c36d-4e16-bb64-cb1697e320ce","Type":"ContainerDied","Data":"992cdc61084516b45a06b69be3065694057b8baf9e8708d3aefa549f983c4e75"}
Nov 02 13:13:05.374684 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.374334    4464 scope.go:122] "RemoveContainer" containerID="000b755a77737abf603701afea6768f6b80c36c01ea607333d61a81dc50f8198"
Nov 02 13:13:05.375859 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.375835    4464 generic.go:356] "Generic (PLEG): container finished" podID="a5f04ed7-b132-4c5a-b86c-07abafe7364a" containerID="258c896b0b87d77c257e1bbcb1a9a85b473738ca92eb8df50975041c6434c4f8" exitCode=0
Nov 02 13:13:05.376010 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.375890    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4" event={"ID":"a5f04ed7-b132-4c5a-b86c-07abafe7364a","Type":"ContainerDied","Data":"258c896b0b87d77c257e1bbcb1a9a85b473738ca92eb8df50975041c6434c4f8"}
Nov 02 13:13:05.376010 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.375908    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"
Nov 02 13:13:05.376010 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.375935    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4" event={"ID":"a5f04ed7-b132-4c5a-b86c-07abafe7364a","Type":"ContainerDied","Data":"858a66df812fcaa45a96a83d8a3a415383a8ab0bf42fe2f04eab9f1ae6965b1a"}
Nov 02 13:13:05.385460 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.385437    4464 scope.go:122] "RemoveContainer" containerID="000b755a77737abf603701afea6768f6b80c36c01ea607333d61a81dc50f8198"
Nov 02 13:13:05.385745 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:05.385724    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"000b755a77737abf603701afea6768f6b80c36c01ea607333d61a81dc50f8198\": container with ID starting with 000b755a77737abf603701afea6768f6b80c36c01ea607333d61a81dc50f8198 not found: ID does not exist" containerID="000b755a77737abf603701afea6768f6b80c36c01ea607333d61a81dc50f8198"
Nov 02 13:13:05.385819 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.385756    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"000b755a77737abf603701afea6768f6b80c36c01ea607333d61a81dc50f8198"} err="failed to get container status \"000b755a77737abf603701afea6768f6b80c36c01ea607333d61a81dc50f8198\": rpc error: code = NotFound desc = could not find container \"000b755a77737abf603701afea6768f6b80c36c01ea607333d61a81dc50f8198\": container with ID starting with 000b755a77737abf603701afea6768f6b80c36c01ea607333d61a81dc50f8198 not found: ID does not exist"
Nov 02 13:13:05.385819 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.385779    4464 scope.go:122] "RemoveContainer" containerID="258c896b0b87d77c257e1bbcb1a9a85b473738ca92eb8df50975041c6434c4f8"
Nov 02 13:13:05.393472 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.393456    4464 scope.go:122] "RemoveContainer" containerID="258c896b0b87d77c257e1bbcb1a9a85b473738ca92eb8df50975041c6434c4f8"
Nov 02 13:13:05.393732 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:05.393711    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258c896b0b87d77c257e1bbcb1a9a85b473738ca92eb8df50975041c6434c4f8\": container with ID starting with 258c896b0b87d77c257e1bbcb1a9a85b473738ca92eb8df50975041c6434c4f8 not found: ID does not exist" containerID="258c896b0b87d77c257e1bbcb1a9a85b473738ca92eb8df50975041c6434c4f8"
Nov 02 13:13:05.393791 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.393739    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258c896b0b87d77c257e1bbcb1a9a85b473738ca92eb8df50975041c6434c4f8"} err="failed to get container status \"258c896b0b87d77c257e1bbcb1a9a85b473738ca92eb8df50975041c6434c4f8\": rpc error: code = NotFound desc = could not find container \"258c896b0b87d77c257e1bbcb1a9a85b473738ca92eb8df50975041c6434c4f8\": container with ID starting with 258c896b0b87d77c257e1bbcb1a9a85b473738ca92eb8df50975041c6434c4f8 not found: ID does not exist"
Nov 02 13:13:05.541221 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.541186    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"]
Nov 02 13:13:05.591884 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.591852    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-6655z59k4"]
Nov 02 13:13:05.655932 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.655850    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"]
Nov 02 13:13:05.660399 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:05.660373    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6ac03f/payload-processing-external-model-e2e-extmod-a-6ac03f-c47dxfx4w"]
Nov 02 13:13:07.353083 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.353046    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b" podUID="5726a503-7324-4dd0-b063-2b9d820d63c3" containerName="payload-processing" containerID="cri-o://c63c7f03b1afe9c00cd9acd8e96db8a96f666ca23e85eacb0772ea378101effc" gracePeriod=25
Nov 02 13:13:07.404189 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.404156    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx" podUID="0c8eebe5-2c24-49c9-abd3-2890b0969071" containerName="payload-processing" containerID="cri-o://4ba05b6da2b4cce4eb7333380e8f1d849bce6d26671cfe5977e600595c6273a9" gracePeriod=25
Nov 02 13:13:07.582086 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.582055    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4696e4-c36d-4e16-bb64-cb1697e320ce" path="/var/lib/kubelet/pods/7b4696e4-c36d-4e16-bb64-cb1697e320ce/volumes"
Nov 02 13:13:07.582498 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.582484    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f04ed7-b132-4c5a-b86c-07abafe7364a" path="/var/lib/kubelet/pods/a5f04ed7-b132-4c5a-b86c-07abafe7364a/volumes"
Nov 02 13:13:07.644459 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.644434    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"
Nov 02 13:13:07.676255 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.676225    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:13:07.696411 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.696388    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c8eebe5-2c24-49c9-abd3-2890b0969071-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c8eebe5-2c24-49c9-abd3-2890b0969071-plugins-config-volume\") pod \"0c8eebe5-2c24-49c9-abd3-2890b0969071\" (UID: \"0c8eebe5-2c24-49c9-abd3-2890b0969071\") "
Nov 02 13:13:07.696535 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.696424    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c8eebe5-2c24-49c9-abd3-2890b0969071-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c8eebe5-2c24-49c9-abd3-2890b0969071-provider-credentials\") pod \"0c8eebe5-2c24-49c9-abd3-2890b0969071\" (UID: \"0c8eebe5-2c24-49c9-abd3-2890b0969071\") "
Nov 02 13:13:07.696535 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.696444    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5726a503-7324-4dd0-b063-2b9d820d63c3-kube-api-access-4lrvv\" (UniqueName: \"kubernetes.io/projected/5726a503-7324-4dd0-b063-2b9d820d63c3-kube-api-access-4lrvv\") pod \"5726a503-7324-4dd0-b063-2b9d820d63c3\" (UID: \"5726a503-7324-4dd0-b063-2b9d820d63c3\") "
Nov 02 13:13:07.696622 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.696576    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5726a503-7324-4dd0-b063-2b9d820d63c3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5726a503-7324-4dd0-b063-2b9d820d63c3-provider-credentials\") pod \"5726a503-7324-4dd0-b063-2b9d820d63c3\" (UID: \"5726a503-7324-4dd0-b063-2b9d820d63c3\") "
Nov 02 13:13:07.696679 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.696647    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c8eebe5-2c24-49c9-abd3-2890b0969071-kube-api-access-fp7xc\" (UniqueName: \"kubernetes.io/projected/0c8eebe5-2c24-49c9-abd3-2890b0969071-kube-api-access-fp7xc\") pod \"0c8eebe5-2c24-49c9-abd3-2890b0969071\" (UID: \"0c8eebe5-2c24-49c9-abd3-2890b0969071\") "
Nov 02 13:13:07.696740 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.696677    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5726a503-7324-4dd0-b063-2b9d820d63c3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5726a503-7324-4dd0-b063-2b9d820d63c3-routing-overlay-volume\") pod \"5726a503-7324-4dd0-b063-2b9d820d63c3\" (UID: \"5726a503-7324-4dd0-b063-2b9d820d63c3\") "
Nov 02 13:13:07.696811 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.696775    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c8eebe5-2c24-49c9-abd3-2890b0969071-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c8eebe5-2c24-49c9-abd3-2890b0969071-routing-overlay-volume\") pod \"0c8eebe5-2c24-49c9-abd3-2890b0969071\" (UID: \"0c8eebe5-2c24-49c9-abd3-2890b0969071\") "
Nov 02 13:13:07.696869 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.696801    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c8eebe5-2c24-49c9-abd3-2890b0969071-plugins-config-volume" pod "0c8eebe5-2c24-49c9-abd3-2890b0969071" (UID: "0c8eebe5-2c24-49c9-abd3-2890b0969071"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:07.697370 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.697138    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c8eebe5-2c24-49c9-abd3-2890b0969071-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:07.697370 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.697307    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5726a503-7324-4dd0-b063-2b9d820d63c3-routing-overlay-volume" pod "5726a503-7324-4dd0-b063-2b9d820d63c3" (UID: "5726a503-7324-4dd0-b063-2b9d820d63c3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:07.697370 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.697321    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c8eebe5-2c24-49c9-abd3-2890b0969071-routing-overlay-volume" pod "0c8eebe5-2c24-49c9-abd3-2890b0969071" (UID: "0c8eebe5-2c24-49c9-abd3-2890b0969071"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:07.698824 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.698704    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5726a503-7324-4dd0-b063-2b9d820d63c3-kube-api-access-4lrvv" pod "5726a503-7324-4dd0-b063-2b9d820d63c3" (UID: "5726a503-7324-4dd0-b063-2b9d820d63c3"). InnerVolumeSpecName "kube-api-access-4lrvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:07.698824 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.698806    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c8eebe5-2c24-49c9-abd3-2890b0969071-kube-api-access-fp7xc" pod "0c8eebe5-2c24-49c9-abd3-2890b0969071" (UID: "0c8eebe5-2c24-49c9-abd3-2890b0969071"). InnerVolumeSpecName "kube-api-access-fp7xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:07.699193 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.699167    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5726a503-7324-4dd0-b063-2b9d820d63c3-provider-credentials" pod "5726a503-7324-4dd0-b063-2b9d820d63c3" (UID: "5726a503-7324-4dd0-b063-2b9d820d63c3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:07.699193 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.699172    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c8eebe5-2c24-49c9-abd3-2890b0969071-provider-credentials" pod "0c8eebe5-2c24-49c9-abd3-2890b0969071" (UID: "0c8eebe5-2c24-49c9-abd3-2890b0969071"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:07.797478 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.797452    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5726a503-7324-4dd0-b063-2b9d820d63c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5726a503-7324-4dd0-b063-2b9d820d63c3-plugins-config-volume\") pod \"5726a503-7324-4dd0-b063-2b9d820d63c3\" (UID: \"5726a503-7324-4dd0-b063-2b9d820d63c3\") "
Nov 02 13:13:07.797672 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.797631    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c8eebe5-2c24-49c9-abd3-2890b0969071-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:07.797672 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.797642    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lrvv\" (UniqueName: \"kubernetes.io/projected/5726a503-7324-4dd0-b063-2b9d820d63c3-kube-api-access-4lrvv\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:07.797672 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.797651    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5726a503-7324-4dd0-b063-2b9d820d63c3-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:07.797672 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.797659    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp7xc\" (UniqueName: \"kubernetes.io/projected/0c8eebe5-2c24-49c9-abd3-2890b0969071-kube-api-access-fp7xc\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:07.797672 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.797669    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5726a503-7324-4dd0-b063-2b9d820d63c3-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:07.797952 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.797679    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c8eebe5-2c24-49c9-abd3-2890b0969071-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:07.797952 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.797871    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5726a503-7324-4dd0-b063-2b9d820d63c3-plugins-config-volume" pod "5726a503-7324-4dd0-b063-2b9d820d63c3" (UID: "5726a503-7324-4dd0-b063-2b9d820d63c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:07.898303 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:07.898232    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5726a503-7324-4dd0-b063-2b9d820d63c3-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:08.394360 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.394326    4464 generic.go:356] "Generic (PLEG): container finished" podID="5726a503-7324-4dd0-b063-2b9d820d63c3" containerID="c63c7f03b1afe9c00cd9acd8e96db8a96f666ca23e85eacb0772ea378101effc" exitCode=0
Nov 02 13:13:08.394805 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.394382    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"
Nov 02 13:13:08.394805 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.394410    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b" event={"ID":"5726a503-7324-4dd0-b063-2b9d820d63c3","Type":"ContainerDied","Data":"c63c7f03b1afe9c00cd9acd8e96db8a96f666ca23e85eacb0772ea378101effc"}
Nov 02 13:13:08.394805 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.394472    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b" event={"ID":"5726a503-7324-4dd0-b063-2b9d820d63c3","Type":"ContainerDied","Data":"05c447b46b772bb589c3b41b10355aba7454e4ff77ae9d3deb16eca64dc3534d"}
Nov 02 13:13:08.394805 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.394503    4464 scope.go:122] "RemoveContainer" containerID="c63c7f03b1afe9c00cd9acd8e96db8a96f666ca23e85eacb0772ea378101effc"
Nov 02 13:13:08.395826 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.395739    4464 generic.go:356] "Generic (PLEG): container finished" podID="0c8eebe5-2c24-49c9-abd3-2890b0969071" containerID="4ba05b6da2b4cce4eb7333380e8f1d849bce6d26671cfe5977e600595c6273a9" exitCode=0
Nov 02 13:13:08.395826 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.395787    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"
Nov 02 13:13:08.395826 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.395796    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx" event={"ID":"0c8eebe5-2c24-49c9-abd3-2890b0969071","Type":"ContainerDied","Data":"4ba05b6da2b4cce4eb7333380e8f1d849bce6d26671cfe5977e600595c6273a9"}
Nov 02 13:13:08.396001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.395836    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx" event={"ID":"0c8eebe5-2c24-49c9-abd3-2890b0969071","Type":"ContainerDied","Data":"6571b3723b7af37e6a8e870027bac23f9412bfb75282c4a0ddad528ceb6dd74c"}
Nov 02 13:13:08.403707 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.403680    4464 scope.go:122] "RemoveContainer" containerID="c63c7f03b1afe9c00cd9acd8e96db8a96f666ca23e85eacb0772ea378101effc"
Nov 02 13:13:08.404169 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:08.404149    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c63c7f03b1afe9c00cd9acd8e96db8a96f666ca23e85eacb0772ea378101effc\": container with ID starting with c63c7f03b1afe9c00cd9acd8e96db8a96f666ca23e85eacb0772ea378101effc not found: ID does not exist" containerID="c63c7f03b1afe9c00cd9acd8e96db8a96f666ca23e85eacb0772ea378101effc"
Nov 02 13:13:08.404239 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.404175    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c63c7f03b1afe9c00cd9acd8e96db8a96f666ca23e85eacb0772ea378101effc"} err="failed to get container status \"c63c7f03b1afe9c00cd9acd8e96db8a96f666ca23e85eacb0772ea378101effc\": rpc error: code = NotFound desc = could not find container \"c63c7f03b1afe9c00cd9acd8e96db8a96f666ca23e85eacb0772ea378101effc\": container with ID starting with c63c7f03b1afe9c00cd9acd8e96db8a96f666ca23e85eacb0772ea378101effc not found: ID does not exist"
Nov 02 13:13:08.404239 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.404190    4464 scope.go:122] "RemoveContainer" containerID="4ba05b6da2b4cce4eb7333380e8f1d849bce6d26671cfe5977e600595c6273a9"
Nov 02 13:13:08.412763 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.412749    4464 scope.go:122] "RemoveContainer" containerID="4ba05b6da2b4cce4eb7333380e8f1d849bce6d26671cfe5977e600595c6273a9"
Nov 02 13:13:08.413021 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:08.412992    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba05b6da2b4cce4eb7333380e8f1d849bce6d26671cfe5977e600595c6273a9\": container with ID starting with 4ba05b6da2b4cce4eb7333380e8f1d849bce6d26671cfe5977e600595c6273a9 not found: ID does not exist" containerID="4ba05b6da2b4cce4eb7333380e8f1d849bce6d26671cfe5977e600595c6273a9"
Nov 02 13:13:08.413100 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.413027    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba05b6da2b4cce4eb7333380e8f1d849bce6d26671cfe5977e600595c6273a9"} err="failed to get container status \"4ba05b6da2b4cce4eb7333380e8f1d849bce6d26671cfe5977e600595c6273a9\": rpc error: code = NotFound desc = could not find container \"4ba05b6da2b4cce4eb7333380e8f1d849bce6d26671cfe5977e600595c6273a9\": container with ID starting with 4ba05b6da2b4cce4eb7333380e8f1d849bce6d26671cfe5977e600595c6273a9 not found: ID does not exist"
Nov 02 13:13:08.562386 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.562357    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"]
Nov 02 13:13:08.568298 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.568275    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8688vv25b"]
Nov 02 13:13:08.678765 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.678690    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"]
Nov 02 13:13:08.682895 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:08.682867    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-b694542hx"]
Nov 02 13:13:09.581044 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:09.581009    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c8eebe5-2c24-49c9-abd3-2890b0969071" path="/var/lib/kubelet/pods/0c8eebe5-2c24-49c9-abd3-2890b0969071/volumes"
Nov 02 13:13:09.581436 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:09.581397    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5726a503-7324-4dd0-b063-2b9d820d63c3" path="/var/lib/kubelet/pods/5726a503-7324-4dd0-b063-2b9d820d63c3/volumes"
Nov 02 13:13:11.489496 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:11.489459    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:13:23.211677 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:23.211641    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:13:27.875280 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:27.875194    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"]
Nov 02 13:13:27.875664 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:27.875410    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2" podUID="90e7a43f-8da1-4a73-ab5e-f5edd3573d80" containerName="istio-proxy" containerID="cri-o://b2289d6fc6cbaf7a0163f006462f31cabec0e3ee92e56a3e9e5af1fdcb85ba2e" gracePeriod=30
Nov 02 13:13:28.291735 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:28.291695    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:13:32.774800 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:32.774766    4464 patch_prober.go:29] interesting pod/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:13:32.775281 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:32.774819    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2" podUID="90e7a43f-8da1-4a73-ab5e-f5edd3573d80" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:13:33.122477 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.122453    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 13:13:33.245811 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.245783    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-data\") pod \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") "
Nov 02 13:13:33.246019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.245823    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-kube-api-access-rprb5\" (UniqueName: \"kubernetes.io/projected/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-kube-api-access-rprb5\") pod \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") "
Nov 02 13:13:33.246019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.245856    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-workload-certs\") pod \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") "
Nov 02 13:13:33.246019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.245873    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-token\" (UniqueName: \"kubernetes.io/projected/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-token\") pod \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") "
Nov 02 13:13:33.246019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.245923    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-podinfo\") pod \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") "
Nov 02 13:13:33.246019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.245949    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-envoy\") pod \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") "
Nov 02 13:13:33.246328 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.246024    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istiod-ca-cert\") pod \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") "
Nov 02 13:13:33.246328 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.246093    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-credential-socket\") pod \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") "
Nov 02 13:13:33.246328 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.246118    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-workload-socket\") pod \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\" (UID: \"90e7a43f-8da1-4a73-ab5e-f5edd3573d80\") "
Nov 02 13:13:33.246328 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.246132    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-data" pod "90e7a43f-8da1-4a73-ab5e-f5edd3573d80" (UID: "90e7a43f-8da1-4a73-ab5e-f5edd3573d80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:13:33.246328 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.246140    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-workload-certs" pod "90e7a43f-8da1-4a73-ab5e-f5edd3573d80" (UID: "90e7a43f-8da1-4a73-ab5e-f5edd3573d80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:13:33.246601 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.246385    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-credential-socket" pod "90e7a43f-8da1-4a73-ab5e-f5edd3573d80" (UID: "90e7a43f-8da1-4a73-ab5e-f5edd3573d80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:13:33.246601 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.246481    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istiod-ca-cert" pod "90e7a43f-8da1-4a73-ab5e-f5edd3573d80" (UID: "90e7a43f-8da1-4a73-ab5e-f5edd3573d80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:33.246601 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.246566    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:33.246601 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.246584    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:33.246601 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.246599    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:33.246887 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.246602    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-workload-socket" pod "90e7a43f-8da1-4a73-ab5e-f5edd3573d80" (UID: "90e7a43f-8da1-4a73-ab5e-f5edd3573d80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:13:33.248297 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.248270    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-kube-api-access-rprb5" pod "90e7a43f-8da1-4a73-ab5e-f5edd3573d80" (UID: "90e7a43f-8da1-4a73-ab5e-f5edd3573d80"). InnerVolumeSpecName "kube-api-access-rprb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:33.248297 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.248288    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-token" pod "90e7a43f-8da1-4a73-ab5e-f5edd3573d80" (UID: "90e7a43f-8da1-4a73-ab5e-f5edd3573d80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:33.248297 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.248270    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-podinfo" pod "90e7a43f-8da1-4a73-ab5e-f5edd3573d80" (UID: "90e7a43f-8da1-4a73-ab5e-f5edd3573d80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:13:33.248537 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.248353    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-envoy" pod "90e7a43f-8da1-4a73-ab5e-f5edd3573d80" (UID: "90e7a43f-8da1-4a73-ab5e-f5edd3573d80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:13:33.347160 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.347080    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rprb5\" (UniqueName: \"kubernetes.io/projected/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-kube-api-access-rprb5\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:33.347160 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.347110    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:33.347160 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.347120    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:33.347160 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.347128    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:33.347160 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.347136    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:33.347160 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.347145    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e7a43f-8da1-4a73-ab5e-f5edd3573d80-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:33.531732 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.531698    4464 generic.go:356] "Generic (PLEG): container finished" podID="90e7a43f-8da1-4a73-ab5e-f5edd3573d80" containerID="b2289d6fc6cbaf7a0163f006462f31cabec0e3ee92e56a3e9e5af1fdcb85ba2e" exitCode=0
Nov 02 13:13:33.531941 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.531754    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"
Nov 02 13:13:33.531941 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.531782    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2" event={"ID":"90e7a43f-8da1-4a73-ab5e-f5edd3573d80","Type":"ContainerDied","Data":"b2289d6fc6cbaf7a0163f006462f31cabec0e3ee92e56a3e9e5af1fdcb85ba2e"}
Nov 02 13:13:33.531941 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.531867    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2" event={"ID":"90e7a43f-8da1-4a73-ab5e-f5edd3573d80","Type":"ContainerDied","Data":"345cf07c6ff7615a2851b6928d75288750cf56baafc063feb9c3b1bde198aa04"}
Nov 02 13:13:33.531941 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.531897    4464 scope.go:122] "RemoveContainer" containerID="b2289d6fc6cbaf7a0163f006462f31cabec0e3ee92e56a3e9e5af1fdcb85ba2e"
Nov 02 13:13:33.541566 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.541550    4464 scope.go:122] "RemoveContainer" containerID="1ff1b5d3ae12317a255583ef40209d3a74e3519e9a2e859154157afb75f9c2b8"
Nov 02 13:13:33.549327 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.549313    4464 scope.go:122] "RemoveContainer" containerID="b2289d6fc6cbaf7a0163f006462f31cabec0e3ee92e56a3e9e5af1fdcb85ba2e"
Nov 02 13:13:33.549582 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:33.549563    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2289d6fc6cbaf7a0163f006462f31cabec0e3ee92e56a3e9e5af1fdcb85ba2e\": container with ID starting with b2289d6fc6cbaf7a0163f006462f31cabec0e3ee92e56a3e9e5af1fdcb85ba2e not found: ID does not exist" containerID="b2289d6fc6cbaf7a0163f006462f31cabec0e3ee92e56a3e9e5af1fdcb85ba2e"
Nov 02 13:13:33.549662 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.549595    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2289d6fc6cbaf7a0163f006462f31cabec0e3ee92e56a3e9e5af1fdcb85ba2e"} err="failed to get container status \"b2289d6fc6cbaf7a0163f006462f31cabec0e3ee92e56a3e9e5af1fdcb85ba2e\": rpc error: code = NotFound desc = could not find container \"b2289d6fc6cbaf7a0163f006462f31cabec0e3ee92e56a3e9e5af1fdcb85ba2e\": container with ID starting with b2289d6fc6cbaf7a0163f006462f31cabec0e3ee92e56a3e9e5af1fdcb85ba2e not found: ID does not exist"
Nov 02 13:13:33.549662 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.549618    4464 scope.go:122] "RemoveContainer" containerID="1ff1b5d3ae12317a255583ef40209d3a74e3519e9a2e859154157afb75f9c2b8"
Nov 02 13:13:33.549890 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:33.549870    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff1b5d3ae12317a255583ef40209d3a74e3519e9a2e859154157afb75f9c2b8\": container with ID starting with 1ff1b5d3ae12317a255583ef40209d3a74e3519e9a2e859154157afb75f9c2b8 not found: ID does not exist" containerID="1ff1b5d3ae12317a255583ef40209d3a74e3519e9a2e859154157afb75f9c2b8"
Nov 02 13:13:33.549930 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.549897    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff1b5d3ae12317a255583ef40209d3a74e3519e9a2e859154157afb75f9c2b8"} err="failed to get container status \"1ff1b5d3ae12317a255583ef40209d3a74e3519e9a2e859154157afb75f9c2b8\": rpc error: code = NotFound desc = could not find container \"1ff1b5d3ae12317a255583ef40209d3a74e3519e9a2e859154157afb75f9c2b8\": container with ID starting with 1ff1b5d3ae12317a255583ef40209d3a74e3519e9a2e859154157afb75f9c2b8 not found: ID does not exist"
Nov 02 13:13:33.721174 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.721137    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"]
Nov 02 13:13:33.771802 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:33.771770    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-6ac03f-openshift-default-568485c58f-29fh2"]
Nov 02 13:13:35.476242 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:35.476209    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"]
Nov 02 13:13:35.580810 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:35.580769    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e7a43f-8da1-4a73-ab5e-f5edd3573d80" path="/var/lib/kubelet/pods/90e7a43f-8da1-4a73-ab5e-f5edd3573d80/volumes"
Nov 02 13:13:35.581168 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:35.581148    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"]
Nov 02 13:13:40.476966 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:40.476906    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9" podUID="9f8d75b7-8678-41a1-bbe0-f14a7107d9b8" containerName="payload-processing" containerID="cri-o://d845c72c2811d7f07d5a08b99cdd1690b80d7d69ba7c3384233e8f07ae4e5ec0" gracePeriod=25
Nov 02 13:13:40.582343 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:40.582292    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk" podUID="ab236a0c-bd75-403b-8563-78617f65229e" containerName="payload-pre-processing" containerID="cri-o://4e49ebae3549223ced3777afd27d7c246d8c979a7d774905e46b598679a3631e" gracePeriod=25
Nov 02 13:13:43.105828 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:43.105794    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:13:47.028248 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:47.028208    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9" podUID="9f8d75b7-8678-41a1-bbe0-f14a7107d9b8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:13:47.028248 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:47.028231    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk" podUID="ab236a0c-bd75-403b-8563-78617f65229e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:13:49.247154 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:49.247118    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"]
Nov 02 13:13:49.247629 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:49.247418    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m" podUID="86b728ba-2cb4-4e99-88b3-d6f0afd94f47" containerName="istio-proxy" containerID="cri-o://7f2a9979275bacef5dbf0e995bc2d32cafc993f294bcc5bfd0777be0957aecee" gracePeriod=30
Nov 02 13:13:52.764813 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:52.764782    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-v6wvs_89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41/limitador/0.log"
Nov 02 13:13:54.017808 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.017776    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-dcd84bcbc-k6pdq_17f4db95-d03a-4200-8b01-d7760251e7fa/manager/0.log"
Nov 02 13:13:54.254441 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.254407    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-28669_5d927c4f-316f-441e-bf28-dfbae4d88bfd/manager/0.log"
Nov 02 13:13:54.496452 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.496429    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 13:13:54.546909 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.546845    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-token\" (UniqueName: \"kubernetes.io/projected/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-token\") pod \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") "
Nov 02 13:13:54.546909 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.546885    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-credential-socket\") pod \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") "
Nov 02 13:13:54.547147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.546925    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-podinfo\") pod \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") "
Nov 02 13:13:54.547147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.546963    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-data\") pod \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") "
Nov 02 13:13:54.547147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.547015    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-kube-api-access-mfm6r\" (UniqueName: \"kubernetes.io/projected/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-kube-api-access-mfm6r\") pod \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") "
Nov 02 13:13:54.547147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.547062    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istiod-ca-cert\") pod \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") "
Nov 02 13:13:54.547147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.547113    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-workload-certs\") pod \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") "
Nov 02 13:13:54.547147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.547143    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-workload-socket\") pod \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") "
Nov 02 13:13:54.547475 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.547191    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-envoy\") pod \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\" (UID: \"86b728ba-2cb4-4e99-88b3-d6f0afd94f47\") "
Nov 02 13:13:54.547475 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.547260    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-credential-socket" pod "86b728ba-2cb4-4e99-88b3-d6f0afd94f47" (UID: "86b728ba-2cb4-4e99-88b3-d6f0afd94f47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:13:54.547475 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.547269    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-data" pod "86b728ba-2cb4-4e99-88b3-d6f0afd94f47" (UID: "86b728ba-2cb4-4e99-88b3-d6f0afd94f47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:13:54.547668 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.547472    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-workload-socket" pod "86b728ba-2cb4-4e99-88b3-d6f0afd94f47" (UID: "86b728ba-2cb4-4e99-88b3-d6f0afd94f47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:13:54.547668 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.547484    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-workload-certs" pod "86b728ba-2cb4-4e99-88b3-d6f0afd94f47" (UID: "86b728ba-2cb4-4e99-88b3-d6f0afd94f47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:13:54.547668 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.547502    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istiod-ca-cert" pod "86b728ba-2cb4-4e99-88b3-d6f0afd94f47" (UID: "86b728ba-2cb4-4e99-88b3-d6f0afd94f47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:54.547668 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.547616    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:54.547668 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.547638    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:54.547668 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.547653    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:54.547668 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.547667    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:54.547969 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.547681    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:54.549282 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.549254    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-kube-api-access-mfm6r" pod "86b728ba-2cb4-4e99-88b3-d6f0afd94f47" (UID: "86b728ba-2cb4-4e99-88b3-d6f0afd94f47"). InnerVolumeSpecName "kube-api-access-mfm6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:54.549486 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.549465    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-podinfo" pod "86b728ba-2cb4-4e99-88b3-d6f0afd94f47" (UID: "86b728ba-2cb4-4e99-88b3-d6f0afd94f47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:13:54.549555 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.549480    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-token" pod "86b728ba-2cb4-4e99-88b3-d6f0afd94f47" (UID: "86b728ba-2cb4-4e99-88b3-d6f0afd94f47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:54.549626 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.549590    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-envoy" pod "86b728ba-2cb4-4e99-88b3-d6f0afd94f47" (UID: "86b728ba-2cb4-4e99-88b3-d6f0afd94f47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:13:54.641017 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.640960    4464 generic.go:356] "Generic (PLEG): container finished" podID="86b728ba-2cb4-4e99-88b3-d6f0afd94f47" containerID="7f2a9979275bacef5dbf0e995bc2d32cafc993f294bcc5bfd0777be0957aecee" exitCode=0
Nov 02 13:13:54.641017 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.641011    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m" event={"ID":"86b728ba-2cb4-4e99-88b3-d6f0afd94f47","Type":"ContainerDied","Data":"7f2a9979275bacef5dbf0e995bc2d32cafc993f294bcc5bfd0777be0957aecee"}
Nov 02 13:13:54.641272 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.641033    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"
Nov 02 13:13:54.641272 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.641064    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m" event={"ID":"86b728ba-2cb4-4e99-88b3-d6f0afd94f47","Type":"ContainerDied","Data":"856408efffdfecedf42b7029cb3716cfcb97150f7cede905ef35532fa66b281f"}
Nov 02 13:13:54.641272 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.641086    4464 scope.go:122] "RemoveContainer" containerID="7f2a9979275bacef5dbf0e995bc2d32cafc993f294bcc5bfd0777be0957aecee"
Nov 02 13:13:54.648585 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.648564    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfm6r\" (UniqueName: \"kubernetes.io/projected/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-kube-api-access-mfm6r\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:54.648585 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.648583    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:54.648766 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.648593    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:54.648766 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.648604    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b728ba-2cb4-4e99-88b3-d6f0afd94f47-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:13:54.651392 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.651374    4464 scope.go:122] "RemoveContainer" containerID="f7b4fc003c3691b7cd85a3f2dd67b9a37e88606e966e6fec8798cbf4e8023b35"
Nov 02 13:13:54.659068 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.659051    4464 scope.go:122] "RemoveContainer" containerID="7f2a9979275bacef5dbf0e995bc2d32cafc993f294bcc5bfd0777be0957aecee"
Nov 02 13:13:54.659318 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:54.659300    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2a9979275bacef5dbf0e995bc2d32cafc993f294bcc5bfd0777be0957aecee\": container with ID starting with 7f2a9979275bacef5dbf0e995bc2d32cafc993f294bcc5bfd0777be0957aecee not found: ID does not exist" containerID="7f2a9979275bacef5dbf0e995bc2d32cafc993f294bcc5bfd0777be0957aecee"
Nov 02 13:13:54.659359 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.659328    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2a9979275bacef5dbf0e995bc2d32cafc993f294bcc5bfd0777be0957aecee"} err="failed to get container status \"7f2a9979275bacef5dbf0e995bc2d32cafc993f294bcc5bfd0777be0957aecee\": rpc error: code = NotFound desc = could not find container \"7f2a9979275bacef5dbf0e995bc2d32cafc993f294bcc5bfd0777be0957aecee\": container with ID starting with 7f2a9979275bacef5dbf0e995bc2d32cafc993f294bcc5bfd0777be0957aecee not found: ID does not exist"
Nov 02 13:13:54.659359 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.659345    4464 scope.go:122] "RemoveContainer" containerID="f7b4fc003c3691b7cd85a3f2dd67b9a37e88606e966e6fec8798cbf4e8023b35"
Nov 02 13:13:54.659573 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:54.659558    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b4fc003c3691b7cd85a3f2dd67b9a37e88606e966e6fec8798cbf4e8023b35\": container with ID starting with f7b4fc003c3691b7cd85a3f2dd67b9a37e88606e966e6fec8798cbf4e8023b35 not found: ID does not exist" containerID="f7b4fc003c3691b7cd85a3f2dd67b9a37e88606e966e6fec8798cbf4e8023b35"
Nov 02 13:13:54.659615 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.659580    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b4fc003c3691b7cd85a3f2dd67b9a37e88606e966e6fec8798cbf4e8023b35"} err="failed to get container status \"f7b4fc003c3691b7cd85a3f2dd67b9a37e88606e966e6fec8798cbf4e8023b35\": rpc error: code = NotFound desc = could not find container \"f7b4fc003c3691b7cd85a3f2dd67b9a37e88606e966e6fec8798cbf4e8023b35\": container with ID starting with f7b4fc003c3691b7cd85a3f2dd67b9a37e88606e966e6fec8798cbf4e8023b35 not found: ID does not exist"
Nov 02 13:13:54.782356 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.782326    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"]
Nov 02 13:13:54.786690 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:54.786666    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-8x92m"]
Nov 02 13:13:55.411139 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:55.411110    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5975b94bc-pp42k_3e629e5b-98b5-4987-83a2-dc5c8dfcb874/manager/1.log"
Nov 02 13:13:55.581366 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:55.581327    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86b728ba-2cb4-4e99-88b3-d6f0afd94f47" path="/var/lib/kubelet/pods/86b728ba-2cb4-4e99-88b3-d6f0afd94f47/volumes"
Nov 02 13:13:55.867313 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:55.867286    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-6sbtx_5c5948cf-a3f1-4f12-b4ac-073db02a1911/manager/0.log"
Nov 02 13:13:56.128175 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:56.128099    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4z4sj_98f4686d-b894-4e32-91bb-479a67c05b1d/proxy/0.log"
Nov 02 13:13:56.762851 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:56.762823    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5975b94bc-pp42k_3e629e5b-98b5-4987-83a2-dc5c8dfcb874/manager/0.log"
Nov 02 13:13:57.028372 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:57.028282    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk" podUID="ab236a0c-bd75-403b-8563-78617f65229e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:13:57.028537 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:13:57.028282    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9" podUID="9f8d75b7-8678-41a1-bbe0-f14a7107d9b8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:13:58.290850 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:13:58.290813    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:14:00.675757 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:00.675706    4464 generic.go:356] "Generic (PLEG): container finished" podID="9f8d75b7-8678-41a1-bbe0-f14a7107d9b8" containerID="d845c72c2811d7f07d5a08b99cdd1690b80d7d69ba7c3384233e8f07ae4e5ec0" exitCode=0
Nov 02 13:14:00.676272 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:00.675778    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9" event={"ID":"9f8d75b7-8678-41a1-bbe0-f14a7107d9b8","Type":"ContainerDied","Data":"d845c72c2811d7f07d5a08b99cdd1690b80d7d69ba7c3384233e8f07ae4e5ec0"}
Nov 02 13:14:00.734558 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:00.734492    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"
Nov 02 13:14:00.800209 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:00.800180    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f8d75b7-8678-41a1-bbe0-f14a7107d9b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f8d75b7-8678-41a1-bbe0-f14a7107d9b8-plugins-config-volume\") pod \"9f8d75b7-8678-41a1-bbe0-f14a7107d9b8\" (UID: \"9f8d75b7-8678-41a1-bbe0-f14a7107d9b8\") "
Nov 02 13:14:00.800413 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:00.800255    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f8d75b7-8678-41a1-bbe0-f14a7107d9b8-kube-api-access-8xwdt\" (UniqueName: \"kubernetes.io/projected/9f8d75b7-8678-41a1-bbe0-f14a7107d9b8-kube-api-access-8xwdt\") pod \"9f8d75b7-8678-41a1-bbe0-f14a7107d9b8\" (UID: \"9f8d75b7-8678-41a1-bbe0-f14a7107d9b8\") "
Nov 02 13:14:00.800627 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:00.800601    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f8d75b7-8678-41a1-bbe0-f14a7107d9b8-plugins-config-volume" pod "9f8d75b7-8678-41a1-bbe0-f14a7107d9b8" (UID: "9f8d75b7-8678-41a1-bbe0-f14a7107d9b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:14:00.802330 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:00.802308    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8d75b7-8678-41a1-bbe0-f14a7107d9b8-kube-api-access-8xwdt" pod "9f8d75b7-8678-41a1-bbe0-f14a7107d9b8" (UID: "9f8d75b7-8678-41a1-bbe0-f14a7107d9b8"). InnerVolumeSpecName "kube-api-access-8xwdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:14:00.818923 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:00.818902    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"
Nov 02 13:14:00.901568 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:00.901534    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab236a0c-bd75-403b-8563-78617f65229e-kube-api-access-5f65m\" (UniqueName: \"kubernetes.io/projected/ab236a0c-bd75-403b-8563-78617f65229e-kube-api-access-5f65m\") pod \"ab236a0c-bd75-403b-8563-78617f65229e\" (UID: \"ab236a0c-bd75-403b-8563-78617f65229e\") "
Nov 02 13:14:00.901717 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:00.901675    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab236a0c-bd75-403b-8563-78617f65229e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab236a0c-bd75-403b-8563-78617f65229e-plugins-config-volume\") pod \"ab236a0c-bd75-403b-8563-78617f65229e\" (UID: \"ab236a0c-bd75-403b-8563-78617f65229e\") "
Nov 02 13:14:00.901835 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:00.901824    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f8d75b7-8678-41a1-bbe0-f14a7107d9b8-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:14:00.901876 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:00.901838    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xwdt\" (UniqueName: \"kubernetes.io/projected/9f8d75b7-8678-41a1-bbe0-f14a7107d9b8-kube-api-access-8xwdt\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:14:00.902081 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:00.902061    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab236a0c-bd75-403b-8563-78617f65229e-plugins-config-volume" pod "ab236a0c-bd75-403b-8563-78617f65229e" (UID: "ab236a0c-bd75-403b-8563-78617f65229e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:14:00.903658 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:00.903638    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab236a0c-bd75-403b-8563-78617f65229e-kube-api-access-5f65m" pod "ab236a0c-bd75-403b-8563-78617f65229e" (UID: "ab236a0c-bd75-403b-8563-78617f65229e"). InnerVolumeSpecName "kube-api-access-5f65m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:14:01.002946 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.002861    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab236a0c-bd75-403b-8563-78617f65229e-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:14:01.002946 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.002891    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f65m\" (UniqueName: \"kubernetes.io/projected/ab236a0c-bd75-403b-8563-78617f65229e-kube-api-access-5f65m\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:14:01.277650 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.277544    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"]
Nov 02 13:14:01.278016 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.277998    4464 cpu_manager.go:414] "containerMap: removing container" podUID="0c8eebe5-2c24-49c9-abd3-2890b0969071" containerName="payload-processing"
Nov 02 13:14:01.278129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278017    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c8eebe5-2c24-49c9-abd3-2890b0969071" containerName="payload-processing"
Nov 02 13:14:01.278129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278036    4464 cpu_manager.go:414] "containerMap: removing container" podUID="9f8d75b7-8678-41a1-bbe0-f14a7107d9b8" containerName="payload-processing"
Nov 02 13:14:01.278129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278041    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f8d75b7-8678-41a1-bbe0-f14a7107d9b8" containerName="payload-processing"
Nov 02 13:14:01.278129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278052    4464 cpu_manager.go:414] "containerMap: removing container" podUID="86b728ba-2cb4-4e99-88b3-d6f0afd94f47" containerName="istio-proxy"
Nov 02 13:14:01.278129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278057    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b728ba-2cb4-4e99-88b3-d6f0afd94f47" containerName="istio-proxy"
Nov 02 13:14:01.278129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278072    4464 cpu_manager.go:414] "containerMap: removing container" podUID="a5f04ed7-b132-4c5a-b86c-07abafe7364a" containerName="payload-processing"
Nov 02 13:14:01.278129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278077    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f04ed7-b132-4c5a-b86c-07abafe7364a" containerName="payload-processing"
Nov 02 13:14:01.278129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278083    4464 cpu_manager.go:414] "containerMap: removing container" podUID="4620cb6f-7c27-4e6a-b2a2-caea0b4976f0" containerName="revoke-keys"
Nov 02 13:14:01.278129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278089    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4620cb6f-7c27-4e6a-b2a2-caea0b4976f0" containerName="revoke-keys"
Nov 02 13:14:01.278129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278108    4464 cpu_manager.go:414] "containerMap: removing container" podUID="7b4696e4-c36d-4e16-bb64-cb1697e320ce" containerName="payload-processing"
Nov 02 13:14:01.278129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278113    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4696e4-c36d-4e16-bb64-cb1697e320ce" containerName="payload-processing"
Nov 02 13:14:01.278129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278118    4464 cpu_manager.go:414] "containerMap: removing container" podUID="ab236a0c-bd75-403b-8563-78617f65229e" containerName="payload-pre-processing"
Nov 02 13:14:01.278129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278122    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab236a0c-bd75-403b-8563-78617f65229e" containerName="payload-pre-processing"
Nov 02 13:14:01.278129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278131    4464 cpu_manager.go:414] "containerMap: removing container" podUID="90e7a43f-8da1-4a73-ab5e-f5edd3573d80" containerName="istio-proxy"
Nov 02 13:14:01.278129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278136    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e7a43f-8da1-4a73-ab5e-f5edd3573d80" containerName="istio-proxy"
Nov 02 13:14:01.278129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278147    4464 cpu_manager.go:414] "containerMap: removing container" podUID="5726a503-7324-4dd0-b063-2b9d820d63c3" containerName="payload-processing"
Nov 02 13:14:01.279008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278151    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5726a503-7324-4dd0-b063-2b9d820d63c3" containerName="payload-processing"
Nov 02 13:14:01.279008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278158    4464 cpu_manager.go:414] "containerMap: removing container" podUID="90e7a43f-8da1-4a73-ab5e-f5edd3573d80" containerName="istio-proxy"
Nov 02 13:14:01.279008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278163    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e7a43f-8da1-4a73-ab5e-f5edd3573d80" containerName="istio-proxy"
Nov 02 13:14:01.279008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278227    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4696e4-c36d-4e16-bb64-cb1697e320ce" containerName="payload-processing"
Nov 02 13:14:01.279008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278235    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab236a0c-bd75-403b-8563-78617f65229e" containerName="payload-pre-processing"
Nov 02 13:14:01.279008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278242    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e7a43f-8da1-4a73-ab5e-f5edd3573d80" containerName="istio-proxy"
Nov 02 13:14:01.279008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278250    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b728ba-2cb4-4e99-88b3-d6f0afd94f47" containerName="istio-proxy"
Nov 02 13:14:01.279008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278260    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="5726a503-7324-4dd0-b063-2b9d820d63c3" containerName="payload-processing"
Nov 02 13:14:01.279008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278268    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e7a43f-8da1-4a73-ab5e-f5edd3573d80" containerName="istio-proxy"
Nov 02 13:14:01.279008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278274    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c8eebe5-2c24-49c9-abd3-2890b0969071" containerName="payload-processing"
Nov 02 13:14:01.279008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278282    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f8d75b7-8678-41a1-bbe0-f14a7107d9b8" containerName="payload-processing"
Nov 02 13:14:01.279008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278291    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f04ed7-b132-4c5a-b86c-07abafe7364a" containerName="payload-processing"
Nov 02 13:14:01.279008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.278295    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="4620cb6f-7c27-4e6a-b2a2-caea0b4976f0" containerName="revoke-keys"
Nov 02 13:14:01.281559 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.281541    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.305207 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.305169    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-podinfo\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.305366 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.305287    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-workload-socket\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.305440 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.305370    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg72v\" (UniqueName: \"kubernetes.io/projected/3c567554-60bf-46e3-8c4a-0a443cbeee13-kube-api-access-tg72v\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.305440 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.305416    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-envoy\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.305584 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.305560    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-credential-socket\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.305636 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.305612    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-workload-certs\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.305691 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.305651    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-token\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.305750 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.305709    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c567554-60bf-46e3-8c4a-0a443cbeee13-istiod-ca-cert\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.305814 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.305783    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-data\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.335370 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.335339    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c1934578-openshift-default-dockercfg-qj725\""
Nov 02 13:14:01.343289 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.343260    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"]
Nov 02 13:14:01.406405 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.406367    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg72v\" (UniqueName: \"kubernetes.io/projected/3c567554-60bf-46e3-8c4a-0a443cbeee13-kube-api-access-tg72v\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.406598 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.406416    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-envoy\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.406598 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.406459    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-credential-socket\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.406598 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.406481    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-workload-certs\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.406805 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.406600    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-token\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.406805 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.406645    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c567554-60bf-46e3-8c4a-0a443cbeee13-istiod-ca-cert\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.406805 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.406720    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-data\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.407001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.406870    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-podinfo\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.407001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.406891    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-credential-socket\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.407001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.406950    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-workload-socket\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.407173 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.406970    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-workload-certs\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.407173 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.407104    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-data\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.407325 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.407305    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-workload-socket\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.407465 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.407442    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c567554-60bf-46e3-8c4a-0a443cbeee13-istiod-ca-cert\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.408770 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.408748    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-envoy\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.409179 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.409162    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-podinfo\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.468836 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.468804    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg72v\" (UniqueName: \"kubernetes.io/projected/3c567554-60bf-46e3-8c4a-0a443cbeee13-kube-api-access-tg72v\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.469105 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.469083    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-token\") pod \"e2e-ait-c1934578-openshift-default-9fd49846f-98hx4\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") " pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.594291 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.594269    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:01.681953 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.681926    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"
Nov 02 13:14:01.681953 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.681945    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9" event={"ID":"9f8d75b7-8678-41a1-bbe0-f14a7107d9b8","Type":"ContainerDied","Data":"a237236bc3caf10e048e5069bf3b2c23938d14c2ef79425b714068b712f5e936"}
Nov 02 13:14:01.682534 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.682032    4464 scope.go:122] "RemoveContainer" containerID="d845c72c2811d7f07d5a08b99cdd1690b80d7d69ba7c3384233e8f07ae4e5ec0"
Nov 02 13:14:01.683535 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.683508    4464 generic.go:356] "Generic (PLEG): container finished" podID="ab236a0c-bd75-403b-8563-78617f65229e" containerID="4e49ebae3549223ced3777afd27d7c246d8c979a7d774905e46b598679a3631e" exitCode=0
Nov 02 13:14:01.683661 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.683551    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"
Nov 02 13:14:01.683661 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.683568    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk" event={"ID":"ab236a0c-bd75-403b-8563-78617f65229e","Type":"ContainerDied","Data":"4e49ebae3549223ced3777afd27d7c246d8c979a7d774905e46b598679a3631e"}
Nov 02 13:14:01.683661 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.683601    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk" event={"ID":"ab236a0c-bd75-403b-8563-78617f65229e","Type":"ContainerDied","Data":"c0021257f06033f4d2007d83bb5e32bf730ba03e610e998550c48153d71fba32"}
Nov 02 13:14:01.690911 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.690892    4464 scope.go:122] "RemoveContainer" containerID="4e49ebae3549223ced3777afd27d7c246d8c979a7d774905e46b598679a3631e"
Nov 02 13:14:01.699748 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.699547    4464 scope.go:122] "RemoveContainer" containerID="4e49ebae3549223ced3777afd27d7c246d8c979a7d774905e46b598679a3631e"
Nov 02 13:14:01.700019 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:14:01.699936    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e49ebae3549223ced3777afd27d7c246d8c979a7d774905e46b598679a3631e\": container with ID starting with 4e49ebae3549223ced3777afd27d7c246d8c979a7d774905e46b598679a3631e not found: ID does not exist" containerID="4e49ebae3549223ced3777afd27d7c246d8c979a7d774905e46b598679a3631e"
Nov 02 13:14:01.700019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.699969    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e49ebae3549223ced3777afd27d7c246d8c979a7d774905e46b598679a3631e"} err="failed to get container status \"4e49ebae3549223ced3777afd27d7c246d8c979a7d774905e46b598679a3631e\": rpc error: code = NotFound desc = could not find container \"4e49ebae3549223ced3777afd27d7c246d8c979a7d774905e46b598679a3631e\": container with ID starting with 4e49ebae3549223ced3777afd27d7c246d8c979a7d774905e46b598679a3631e not found: ID does not exist"
Nov 02 13:14:01.850721 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.850653    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"]
Nov 02 13:14:01.856483 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.856459    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3a78b4-6bdcd74b8c-44fkk"]
Nov 02 13:14:01.967296 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.967264    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"]
Nov 02 13:14:01.971319 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:01.971296    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3a78b4-54f9b5d795-4c8q9"]
Nov 02 13:14:02.023237 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:02.023163    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"]
Nov 02 13:14:02.027907 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:02.027873    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:14:02.028062 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:02.027948    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:14:02.028062 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:02.028030    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:14:02.692062 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:02.692028    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4" event={"ID":"3c567554-60bf-46e3-8c4a-0a443cbeee13","Type":"ContainerStarted","Data":"a3458d8cbe842951a119657f1975d2c8f70ca644ac1697861afd0754a852b15c"}
Nov 02 13:14:02.692532 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:02.692088    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4" event={"ID":"3c567554-60bf-46e3-8c4a-0a443cbeee13","Type":"ContainerStarted","Data":"64047019ad213f87bd9050ae60750de775f4e14fb0ec680d27f5840e41f15111"}
Nov 02 13:14:02.815985 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:02.815916    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4" podStartSLOduration=1.8159026200000001 podStartE2EDuration="1.81590262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:14:02.814850602 +0000 UTC m=+4671.378559012" watchObservedRunningTime="2026-11-02 13:14:02.81590262 +0000 UTC m=+4671.379611043"
Nov 02 13:14:03.580437 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:03.580401    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f8d75b7-8678-41a1-bbe0-f14a7107d9b8" path="/var/lib/kubelet/pods/9f8d75b7-8678-41a1-bbe0-f14a7107d9b8/volumes"
Nov 02 13:14:03.580912 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:03.580891    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab236a0c-bd75-403b-8563-78617f65229e" path="/var/lib/kubelet/pods/ab236a0c-bd75-403b-8563-78617f65229e/volumes"
Nov 02 13:14:03.594818 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:03.594796    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:03.600075 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:03.600053    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:03.697132 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:03.697103    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:03.697917 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:03.697896    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:08.551505 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.551470    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"]
Nov 02 13:14:08.552130 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.552112    4464 cpu_manager.go:414] "containerMap: removing container" podUID="86b728ba-2cb4-4e99-88b3-d6f0afd94f47" containerName="istio-proxy"
Nov 02 13:14:08.552192 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.552131    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b728ba-2cb4-4e99-88b3-d6f0afd94f47" containerName="istio-proxy"
Nov 02 13:14:08.552233 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.552192    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b728ba-2cb4-4e99-88b3-d6f0afd94f47" containerName="istio-proxy"
Nov 02 13:14:08.556293 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.556275    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"
Nov 02 13:14:08.612674 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.612642    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c1934578\""
Nov 02 13:14:08.615121 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.615092    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"]
Nov 02 13:14:08.663806 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.663775    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c1934578-dockercfg-vbmzx\""
Nov 02 13:14:08.677349 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.677317    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k5rm\" (UniqueName: \"kubernetes.io/projected/f6ff8152-8023-4aab-a08b-42a0620c0630-kube-api-access-5k5rm\") pod \"payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb\" (UID: \"f6ff8152-8023-4aab-a08b-42a0620c0630\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"
Nov 02 13:14:08.677485 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.677370    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ff8152-8023-4aab-a08b-42a0620c0630-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb\" (UID: \"f6ff8152-8023-4aab-a08b-42a0620c0630\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"
Nov 02 13:14:08.700392 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.700365    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"]
Nov 02 13:14:08.704058 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.704038    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"
Nov 02 13:14:08.762804 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.762773    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"]
Nov 02 13:14:08.778325 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.778291    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ff8152-8023-4aab-a08b-42a0620c0630-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb\" (UID: \"f6ff8152-8023-4aab-a08b-42a0620c0630\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"
Nov 02 13:14:08.778489 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.778366    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4p9z\" (UniqueName: \"kubernetes.io/projected/51b7a021-b824-41fe-b7c0-bfbd2f27b76f-kube-api-access-r4p9z\") pod \"payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8\" (UID: \"51b7a021-b824-41fe-b7c0-bfbd2f27b76f\") " pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"
Nov 02 13:14:08.778577 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.778552    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k5rm\" (UniqueName: \"kubernetes.io/projected/f6ff8152-8023-4aab-a08b-42a0620c0630-kube-api-access-5k5rm\") pod \"payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb\" (UID: \"f6ff8152-8023-4aab-a08b-42a0620c0630\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"
Nov 02 13:14:08.778643 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.778596    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b7a021-b824-41fe-b7c0-bfbd2f27b76f-plugins-config-volume\") pod \"payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8\" (UID: \"51b7a021-b824-41fe-b7c0-bfbd2f27b76f\") " pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"
Nov 02 13:14:08.779119 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.779092    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ff8152-8023-4aab-a08b-42a0620c0630-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb\" (UID: \"f6ff8152-8023-4aab-a08b-42a0620c0630\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"
Nov 02 13:14:08.841012 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.840926    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k5rm\" (UniqueName: \"kubernetes.io/projected/f6ff8152-8023-4aab-a08b-42a0620c0630-kube-api-access-5k5rm\") pod \"payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb\" (UID: \"f6ff8152-8023-4aab-a08b-42a0620c0630\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"
Nov 02 13:14:08.868748 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.868721    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"
Nov 02 13:14:08.879373 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.879349    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b7a021-b824-41fe-b7c0-bfbd2f27b76f-plugins-config-volume\") pod \"payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8\" (UID: \"51b7a021-b824-41fe-b7c0-bfbd2f27b76f\") " pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"
Nov 02 13:14:08.879497 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.879410    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4p9z\" (UniqueName: \"kubernetes.io/projected/51b7a021-b824-41fe-b7c0-bfbd2f27b76f-kube-api-access-r4p9z\") pod \"payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8\" (UID: \"51b7a021-b824-41fe-b7c0-bfbd2f27b76f\") " pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"
Nov 02 13:14:08.880228 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.880206    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b7a021-b824-41fe-b7c0-bfbd2f27b76f-plugins-config-volume\") pod \"payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8\" (UID: \"51b7a021-b824-41fe-b7c0-bfbd2f27b76f\") " pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"
Nov 02 13:14:08.941895 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:08.941838    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4p9z\" (UniqueName: \"kubernetes.io/projected/51b7a021-b824-41fe-b7c0-bfbd2f27b76f-kube-api-access-r4p9z\") pod \"payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8\" (UID: \"51b7a021-b824-41fe-b7c0-bfbd2f27b76f\") " pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"
Nov 02 13:14:09.017182 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:09.017146    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"
Nov 02 13:14:09.241060 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:09.241028    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"]
Nov 02 13:14:09.241855 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:14:09.241830    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ff8152_8023_4aab_a08b_42a0620c0630.slice/crio-633b10cd03778a0976790ac2b17d858b251e42a6ee892873ee92c1b373287a8e WatchSource:0}: Error finding container 633b10cd03778a0976790ac2b17d858b251e42a6ee892873ee92c1b373287a8e: Status 404 returned error can't find the container with id 633b10cd03778a0976790ac2b17d858b251e42a6ee892873ee92c1b373287a8e
Nov 02 13:14:09.387908 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:09.387881    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"]
Nov 02 13:14:09.389187 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:14:09.389166    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b7a021_b824_41fe_b7c0_bfbd2f27b76f.slice/crio-70619b8461a9892ec95cd0e6cf4ce1ba012b29caa88b636f3b49183a38ad5927 WatchSource:0}: Error finding container 70619b8461a9892ec95cd0e6cf4ce1ba012b29caa88b636f3b49183a38ad5927: Status 404 returned error can't find the container with id 70619b8461a9892ec95cd0e6cf4ce1ba012b29caa88b636f3b49183a38ad5927
Nov 02 13:14:09.728622 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:09.728583    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8" event={"ID":"51b7a021-b824-41fe-b7c0-bfbd2f27b76f","Type":"ContainerStarted","Data":"ca323572d9b3670bcd1945236e2bd65c4f5026d18f7e93728115a4bee105ad44"}
Nov 02 13:14:09.729112 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:09.728722    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"
Nov 02 13:14:09.729112 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:09.728752    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8" event={"ID":"51b7a021-b824-41fe-b7c0-bfbd2f27b76f","Type":"ContainerStarted","Data":"70619b8461a9892ec95cd0e6cf4ce1ba012b29caa88b636f3b49183a38ad5927"}
Nov 02 13:14:09.730108 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:09.730084    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb" event={"ID":"f6ff8152-8023-4aab-a08b-42a0620c0630","Type":"ContainerStarted","Data":"614759d8a0c0bf0d0874b2ed84f4cb7e839268b5a9da51fdf39f047bb5f4c1c6"}
Nov 02 13:14:09.730236 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:09.730129    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb" event={"ID":"f6ff8152-8023-4aab-a08b-42a0620c0630","Type":"ContainerStarted","Data":"633b10cd03778a0976790ac2b17d858b251e42a6ee892873ee92c1b373287a8e"}
Nov 02 13:14:09.730236 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:09.730164    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"
Nov 02 13:14:09.849296 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:09.849243    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8" podStartSLOduration=1.849225461 podStartE2EDuration="1.849225461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:14:09.8477179 +0000 UTC m=+4678.411426308" watchObservedRunningTime="2026-11-02 13:14:09.849225461 +0000 UTC m=+4678.412933870"
Nov 02 13:14:09.967573 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:09.967515    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb" podStartSLOduration=1.967500654 podStartE2EDuration="1.967500654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:14:09.964755516 +0000 UTC m=+4678.528463930" watchObservedRunningTime="2026-11-02 13:14:09.967500654 +0000 UTC m=+4678.531209062"
Nov 02 13:14:19.379011 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:19.378930    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:14:20.738075 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:20.738046    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"
Nov 02 13:14:20.738475 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:20.738116    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"
Nov 02 13:14:28.292184 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:14:28.292148    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:14:28.838203 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:28.838168    4464 generic.go:356] "Generic (PLEG): container finished" podID="3c567554-60bf-46e3-8c4a-0a443cbeee13" containerID="a3458d8cbe842951a119657f1975d2c8f70ca644ac1697861afd0754a852b15c" exitCode=0
Nov 02 13:14:28.838412 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:28.838245    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4" event={"ID":"3c567554-60bf-46e3-8c4a-0a443cbeee13","Type":"ContainerDied","Data":"a3458d8cbe842951a119657f1975d2c8f70ca644ac1697861afd0754a852b15c"}
Nov 02 13:14:28.838673 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:28.838657    4464 scope.go:122] "RemoveContainer" containerID="a3458d8cbe842951a119657f1975d2c8f70ca644ac1697861afd0754a852b15c"
Nov 02 13:14:28.839890 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:28.839860    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:14:28.839956 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:28.839932    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:14:28.840008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:28.839966    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:14:29.845951 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:29.845916    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4" event={"ID":"3c567554-60bf-46e3-8c4a-0a443cbeee13","Type":"ContainerStarted","Data":"9e2e2af4a8cc28f659ecfa2a23aabdf711a6ab0a9ec6929620ee0d0791012d76"}
Nov 02 13:14:30.594577 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:30.594542    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:30.595515 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:30.595490    4464 patch_prober.go:29] interesting pod/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Nov 02 13:14:30.595658 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:30.595552    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4" podUID="3c567554-60bf-46e3-8c4a-0a443cbeee13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Nov 02 13:14:31.594962 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:31.594921    4464 patch_prober.go:29] interesting pod/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Nov 02 13:14:31.595475 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:31.595020    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4" podUID="3c567554-60bf-46e3-8c4a-0a443cbeee13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Nov 02 13:14:32.595098 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:32.595060    4464 patch_prober.go:29] interesting pod/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Nov 02 13:14:32.595501 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:32.595124    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4" podUID="3c567554-60bf-46e3-8c4a-0a443cbeee13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Nov 02 13:14:33.595346 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:33.595311    4464 patch_prober.go:29] interesting pod/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Nov 02 13:14:33.595746 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:33.595369    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4" podUID="3c567554-60bf-46e3-8c4a-0a443cbeee13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Nov 02 13:14:33.697524 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:33.697472    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:34.595526 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:34.595489    4464 patch_prober.go:29] interesting pod/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body=
Nov 02 13:14:34.595949 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:34.595556    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4" podUID="3c567554-60bf-46e3-8c4a-0a443cbeee13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused"
Nov 02 13:14:35.598944 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:35.598914    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:35.599843 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:35.599822    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:14:41.366925 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:41.366887    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"]
Nov 02 13:14:41.371546 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:41.371529    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"
Nov 02 13:14:41.475877 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:41.475839    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:14:41.475877 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:41.475859    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:14:41.475877 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:41.475865    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:14:41.476218 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:41.475953    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:14:41.478181 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:41.478156    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"]
Nov 02 13:14:41.490337 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:41.490314    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1934578-p7zg7\" (UID: \"8bf4a9a7-8d0d-4612-ab59-e0281234fa62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"
Nov 02 13:14:41.490455 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:41.490438    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1934578-p7zg7\" (UID: \"8bf4a9a7-8d0d-4612-ab59-e0281234fa62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"
Nov 02 13:14:41.490498 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:41.490484    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf8q5\" (UniqueName: \"kubernetes.io/projected/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-kube-api-access-wf8q5\") pod \"maas-api-revoke-keys-e2e-ait-c1934578-p7zg7\" (UID: \"8bf4a9a7-8d0d-4612-ab59-e0281234fa62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"
Nov 02 13:14:41.591690 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:41.591657    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1934578-p7zg7\" (UID: \"8bf4a9a7-8d0d-4612-ab59-e0281234fa62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"
Nov 02 13:14:41.591858 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:41.591704    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf8q5\" (UniqueName: \"kubernetes.io/projected/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-kube-api-access-wf8q5\") pod \"maas-api-revoke-keys-e2e-ait-c1934578-p7zg7\" (UID: \"8bf4a9a7-8d0d-4612-ab59-e0281234fa62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"
Nov 02 13:14:41.591858 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:41.591767    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1934578-p7zg7\" (UID: \"8bf4a9a7-8d0d-4612-ab59-e0281234fa62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"
Nov 02 13:14:41.592489 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:41.592468    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1934578-p7zg7\" (UID: \"8bf4a9a7-8d0d-4612-ab59-e0281234fa62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"
Nov 02 13:14:41.594103 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:41.594082    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c1934578-p7zg7\" (UID: \"8bf4a9a7-8d0d-4612-ab59-e0281234fa62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"
Nov 02 13:14:41.653694 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:41.653624    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf8q5\" (UniqueName: \"kubernetes.io/projected/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-kube-api-access-wf8q5\") pod \"maas-api-revoke-keys-e2e-ait-c1934578-p7zg7\" (UID: \"8bf4a9a7-8d0d-4612-ab59-e0281234fa62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"
Nov 02 13:14:41.684762 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:41.684732    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"
Nov 02 13:14:42.064378 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:42.064353    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"]
Nov 02 13:14:42.065604 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:14:42.065578    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf4a9a7_8d0d_4612_ab59_e0281234fa62.slice/crio-555dc680305004d8b56d1fb40be89a4199f2cd405f93d3795169b196d1f366d5 WatchSource:0}: Error finding container 555dc680305004d8b56d1fb40be89a4199f2cd405f93d3795169b196d1f366d5: Status 404 returned error can't find the container with id 555dc680305004d8b56d1fb40be89a4199f2cd405f93d3795169b196d1f366d5
Nov 02 13:14:42.934748 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:42.934713    4464 generic.go:356] "Generic (PLEG): container finished" podID="8bf4a9a7-8d0d-4612-ab59-e0281234fa62" containerID="518c0dd96f63ee881a567312b0f43f96428175b3d9a9f5053be2a4352fab5ca6" exitCode=0
Nov 02 13:14:42.935128 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:42.934794    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7" event={"ID":"8bf4a9a7-8d0d-4612-ab59-e0281234fa62","Type":"ContainerDied","Data":"518c0dd96f63ee881a567312b0f43f96428175b3d9a9f5053be2a4352fab5ca6"}
Nov 02 13:14:42.935128 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:42.934846    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7" event={"ID":"8bf4a9a7-8d0d-4612-ab59-e0281234fa62","Type":"ContainerStarted","Data":"555dc680305004d8b56d1fb40be89a4199f2cd405f93d3795169b196d1f366d5"}
Nov 02 13:14:44.071915 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:44.071880    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"
Nov 02 13:14:44.115284 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:44.115255    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-kube-api-access-wf8q5\" (UniqueName: \"kubernetes.io/projected/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-kube-api-access-wf8q5\") pod \"8bf4a9a7-8d0d-4612-ab59-e0281234fa62\" (UID: \"8bf4a9a7-8d0d-4612-ab59-e0281234fa62\") "
Nov 02 13:14:44.115435 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:44.115341    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-maas-api-openshift-ca\") pod \"8bf4a9a7-8d0d-4612-ab59-e0281234fa62\" (UID: \"8bf4a9a7-8d0d-4612-ab59-e0281234fa62\") "
Nov 02 13:14:44.115435 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:44.115388    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-maas-api-xks-ca\") pod \"8bf4a9a7-8d0d-4612-ab59-e0281234fa62\" (UID: \"8bf4a9a7-8d0d-4612-ab59-e0281234fa62\") "
Nov 02 13:14:44.115761 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:44.115730    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-maas-api-openshift-ca" pod "8bf4a9a7-8d0d-4612-ab59-e0281234fa62" (UID: "8bf4a9a7-8d0d-4612-ab59-e0281234fa62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:14:44.117311 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:44.117293    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-maas-api-xks-ca" pod "8bf4a9a7-8d0d-4612-ab59-e0281234fa62" (UID: "8bf4a9a7-8d0d-4612-ab59-e0281234fa62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:14:44.117438 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:44.117422    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-kube-api-access-wf8q5" pod "8bf4a9a7-8d0d-4612-ab59-e0281234fa62" (UID: "8bf4a9a7-8d0d-4612-ab59-e0281234fa62"). InnerVolumeSpecName "kube-api-access-wf8q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:14:44.217116 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:44.217036    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:14:44.217116 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:44.217075    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:14:44.217116 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:44.217092    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf8q5\" (UniqueName: \"kubernetes.io/projected/8bf4a9a7-8d0d-4612-ab59-e0281234fa62-kube-api-access-wf8q5\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:14:44.945399 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:44.945364    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7" event={"ID":"8bf4a9a7-8d0d-4612-ab59-e0281234fa62","Type":"ContainerDied","Data":"555dc680305004d8b56d1fb40be89a4199f2cd405f93d3795169b196d1f366d5"}
Nov 02 13:14:44.945579 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:44.945403    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"
Nov 02 13:14:44.945579 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:44.945414    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="555dc680305004d8b56d1fb40be89a4199f2cd405f93d3795169b196d1f366d5"
Nov 02 13:14:52.885234 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:52.885195    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"]
Nov 02 13:14:52.992412 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:52.992380    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"]
Nov 02 13:14:57.885906 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:57.885814    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8" podUID="51b7a021-b824-41fe-b7c0-bfbd2f27b76f" containerName="payload-processing" containerID="cri-o://ca323572d9b3670bcd1945236e2bd65c4f5026d18f7e93728115a4bee105ad44" gracePeriod=25
Nov 02 13:14:57.992819 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:57.992764    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb" podUID="f6ff8152-8023-4aab-a08b-42a0620c0630" containerName="payload-pre-processing" containerID="cri-o://614759d8a0c0bf0d0874b2ed84f4cb7e839268b5a9da51fdf39f047bb5f4c1c6" gracePeriod=25
Nov 02 13:14:58.013903 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:58.013872    4464 generic.go:356] "Generic (PLEG): container finished" podID="51b7a021-b824-41fe-b7c0-bfbd2f27b76f" containerID="ca323572d9b3670bcd1945236e2bd65c4f5026d18f7e93728115a4bee105ad44" exitCode=0
Nov 02 13:14:58.014060 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:58.013944    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8" event={"ID":"51b7a021-b824-41fe-b7c0-bfbd2f27b76f","Type":"ContainerDied","Data":"ca323572d9b3670bcd1945236e2bd65c4f5026d18f7e93728115a4bee105ad44"}
Nov 02 13:14:58.123208 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:58.123183    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"
Nov 02 13:14:58.237800 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:58.237776    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"
Nov 02 13:14:58.252798 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:58.252763    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b7a021-b824-41fe-b7c0-bfbd2f27b76f-kube-api-access-r4p9z\" (UniqueName: \"kubernetes.io/projected/51b7a021-b824-41fe-b7c0-bfbd2f27b76f-kube-api-access-r4p9z\") pod \"51b7a021-b824-41fe-b7c0-bfbd2f27b76f\" (UID: \"51b7a021-b824-41fe-b7c0-bfbd2f27b76f\") "
Nov 02 13:14:58.252958 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:58.252880    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51b7a021-b824-41fe-b7c0-bfbd2f27b76f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b7a021-b824-41fe-b7c0-bfbd2f27b76f-plugins-config-volume\") pod \"51b7a021-b824-41fe-b7c0-bfbd2f27b76f\" (UID: \"51b7a021-b824-41fe-b7c0-bfbd2f27b76f\") "
Nov 02 13:14:58.253703 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:58.253675    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b7a021-b824-41fe-b7c0-bfbd2f27b76f-plugins-config-volume" pod "51b7a021-b824-41fe-b7c0-bfbd2f27b76f" (UID: "51b7a021-b824-41fe-b7c0-bfbd2f27b76f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:14:58.256201 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:58.256172    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b7a021-b824-41fe-b7c0-bfbd2f27b76f-kube-api-access-r4p9z" pod "51b7a021-b824-41fe-b7c0-bfbd2f27b76f" (UID: "51b7a021-b824-41fe-b7c0-bfbd2f27b76f"). InnerVolumeSpecName "kube-api-access-r4p9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:14:58.291120 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:14:58.291086    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:14:58.354683 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:58.354651    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6ff8152-8023-4aab-a08b-42a0620c0630-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ff8152-8023-4aab-a08b-42a0620c0630-plugins-config-volume\") pod \"f6ff8152-8023-4aab-a08b-42a0620c0630\" (UID: \"f6ff8152-8023-4aab-a08b-42a0620c0630\") "
Nov 02 13:14:58.354884 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:58.354720    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ff8152-8023-4aab-a08b-42a0620c0630-kube-api-access-5k5rm\" (UniqueName: \"kubernetes.io/projected/f6ff8152-8023-4aab-a08b-42a0620c0630-kube-api-access-5k5rm\") pod \"f6ff8152-8023-4aab-a08b-42a0620c0630\" (UID: \"f6ff8152-8023-4aab-a08b-42a0620c0630\") "
Nov 02 13:14:58.355029 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:58.355009    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4p9z\" (UniqueName: \"kubernetes.io/projected/51b7a021-b824-41fe-b7c0-bfbd2f27b76f-kube-api-access-r4p9z\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:14:58.355099 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:58.355035    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51b7a021-b824-41fe-b7c0-bfbd2f27b76f-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:14:58.355141 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:58.355102    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ff8152-8023-4aab-a08b-42a0620c0630-plugins-config-volume" pod "f6ff8152-8023-4aab-a08b-42a0620c0630" (UID: "f6ff8152-8023-4aab-a08b-42a0620c0630"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:14:58.356784 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:58.356766    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ff8152-8023-4aab-a08b-42a0620c0630-kube-api-access-5k5rm" pod "f6ff8152-8023-4aab-a08b-42a0620c0630" (UID: "f6ff8152-8023-4aab-a08b-42a0620c0630"). InnerVolumeSpecName "kube-api-access-5k5rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:14:58.455573 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:58.455492    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k5rm\" (UniqueName: \"kubernetes.io/projected/f6ff8152-8023-4aab-a08b-42a0620c0630-kube-api-access-5k5rm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:14:58.455573 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:58.455520    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ff8152-8023-4aab-a08b-42a0620c0630-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:14:59.020304 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:59.020268    4464 generic.go:356] "Generic (PLEG): container finished" podID="f6ff8152-8023-4aab-a08b-42a0620c0630" containerID="614759d8a0c0bf0d0874b2ed84f4cb7e839268b5a9da51fdf39f047bb5f4c1c6" exitCode=0
Nov 02 13:14:59.020750 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:59.020326    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"
Nov 02 13:14:59.020750 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:59.020362    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb" event={"ID":"f6ff8152-8023-4aab-a08b-42a0620c0630","Type":"ContainerDied","Data":"614759d8a0c0bf0d0874b2ed84f4cb7e839268b5a9da51fdf39f047bb5f4c1c6"}
Nov 02 13:14:59.020750 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:59.020431    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb" event={"ID":"f6ff8152-8023-4aab-a08b-42a0620c0630","Type":"ContainerDied","Data":"633b10cd03778a0976790ac2b17d858b251e42a6ee892873ee92c1b373287a8e"}
Nov 02 13:14:59.020750 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:59.020465    4464 scope.go:122] "RemoveContainer" containerID="614759d8a0c0bf0d0874b2ed84f4cb7e839268b5a9da51fdf39f047bb5f4c1c6"
Nov 02 13:14:59.021847 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:59.021785    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"
Nov 02 13:14:59.021847 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:59.021793    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8" event={"ID":"51b7a021-b824-41fe-b7c0-bfbd2f27b76f","Type":"ContainerDied","Data":"70619b8461a9892ec95cd0e6cf4ce1ba012b29caa88b636f3b49183a38ad5927"}
Nov 02 13:14:59.029622 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:59.029520    4464 scope.go:122] "RemoveContainer" containerID="614759d8a0c0bf0d0874b2ed84f4cb7e839268b5a9da51fdf39f047bb5f4c1c6"
Nov 02 13:14:59.029844 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:14:59.029815    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614759d8a0c0bf0d0874b2ed84f4cb7e839268b5a9da51fdf39f047bb5f4c1c6\": container with ID starting with 614759d8a0c0bf0d0874b2ed84f4cb7e839268b5a9da51fdf39f047bb5f4c1c6 not found: ID does not exist" containerID="614759d8a0c0bf0d0874b2ed84f4cb7e839268b5a9da51fdf39f047bb5f4c1c6"
Nov 02 13:14:59.029931 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:59.029855    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614759d8a0c0bf0d0874b2ed84f4cb7e839268b5a9da51fdf39f047bb5f4c1c6"} err="failed to get container status \"614759d8a0c0bf0d0874b2ed84f4cb7e839268b5a9da51fdf39f047bb5f4c1c6\": rpc error: code = NotFound desc = could not find container \"614759d8a0c0bf0d0874b2ed84f4cb7e839268b5a9da51fdf39f047bb5f4c1c6\": container with ID starting with 614759d8a0c0bf0d0874b2ed84f4cb7e839268b5a9da51fdf39f047bb5f4c1c6 not found: ID does not exist"
Nov 02 13:14:59.029931 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:59.029877    4464 scope.go:122] "RemoveContainer" containerID="ca323572d9b3670bcd1945236e2bd65c4f5026d18f7e93728115a4bee105ad44"
Nov 02 13:14:59.187202 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:59.187120    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"]
Nov 02 13:14:59.237788 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:59.237754    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c1934578-7ccfdc97ff-2phh8"]
Nov 02 13:14:59.348848 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:59.348819    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"]
Nov 02 13:14:59.353322 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:59.353299    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c1934578-5c588b5bdc-s8gfb"]
Nov 02 13:14:59.580514 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:59.580483    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51b7a021-b824-41fe-b7c0-bfbd2f27b76f" path="/var/lib/kubelet/pods/51b7a021-b824-41fe-b7c0-bfbd2f27b76f/volumes"
Nov 02 13:14:59.580934 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:14:59.580914    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6ff8152-8023-4aab-a08b-42a0620c0630" path="/var/lib/kubelet/pods/f6ff8152-8023-4aab-a08b-42a0620c0630/volumes"
Nov 02 13:15:00.176468 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.176430    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz"]
Nov 02 13:15:00.177095 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.177076    4464 cpu_manager.go:414] "containerMap: removing container" podUID="8bf4a9a7-8d0d-4612-ab59-e0281234fa62" containerName="revoke-keys"
Nov 02 13:15:00.177149 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.177098    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf4a9a7-8d0d-4612-ab59-e0281234fa62" containerName="revoke-keys"
Nov 02 13:15:00.177149 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.177118    4464 cpu_manager.go:414] "containerMap: removing container" podUID="51b7a021-b824-41fe-b7c0-bfbd2f27b76f" containerName="payload-processing"
Nov 02 13:15:00.177149 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.177126    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b7a021-b824-41fe-b7c0-bfbd2f27b76f" containerName="payload-processing"
Nov 02 13:15:00.177149 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.177145    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f6ff8152-8023-4aab-a08b-42a0620c0630" containerName="payload-pre-processing"
Nov 02 13:15:00.177296 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.177152    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ff8152-8023-4aab-a08b-42a0620c0630" containerName="payload-pre-processing"
Nov 02 13:15:00.177296 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.177285    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf4a9a7-8d0d-4612-ab59-e0281234fa62" containerName="revoke-keys"
Nov 02 13:15:00.177362 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.177299    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b7a021-b824-41fe-b7c0-bfbd2f27b76f" containerName="payload-processing"
Nov 02 13:15:00.177362 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.177310    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ff8152-8023-4aab-a08b-42a0620c0630" containerName="payload-pre-processing"
Nov 02 13:15:00.182048 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.182031    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc"]
Nov 02 13:15:00.182167 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.182152    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz"
Nov 02 13:15:00.185445 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.185426    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc"
Nov 02 13:15:00.237069 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.237047    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz"]
Nov 02 13:15:00.270898 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.270866    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf272\" (UniqueName: \"kubernetes.io/projected/e71eddf4-e654-4f1b-817e-b6eab12f8571-kube-api-access-vf272\") pod \"maas-api-key-cleanup-29893755-67pcc\" (UID: \"e71eddf4-e654-4f1b-817e-b6eab12f8571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc"
Nov 02 13:15:00.271098 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.270906    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl9kv\" (UniqueName: \"kubernetes.io/projected/75a852d2-c232-4e68-a79a-e945930aef35-kube-api-access-dl9kv\") pod \"maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz\" (UID: \"75a852d2-c232-4e68-a79a-e945930aef35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz"
Nov 02 13:15:00.271098 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.270930    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a852d2-c232-4e68-a79a-e945930aef35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz\" (UID: \"75a852d2-c232-4e68-a79a-e945930aef35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz"
Nov 02 13:15:00.271098 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.271074    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71eddf4-e654-4f1b-817e-b6eab12f8571-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893755-67pcc\" (UID: \"e71eddf4-e654-4f1b-817e-b6eab12f8571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc"
Nov 02 13:15:00.284913 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.284886    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:15:00.285088 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.285069    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:15:00.285201 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.285181    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:15:00.288270 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.288249    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc"]
Nov 02 13:15:00.371742 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.371710    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf272\" (UniqueName: \"kubernetes.io/projected/e71eddf4-e654-4f1b-817e-b6eab12f8571-kube-api-access-vf272\") pod \"maas-api-key-cleanup-29893755-67pcc\" (UID: \"e71eddf4-e654-4f1b-817e-b6eab12f8571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc"
Nov 02 13:15:00.371908 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.371762    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl9kv\" (UniqueName: \"kubernetes.io/projected/75a852d2-c232-4e68-a79a-e945930aef35-kube-api-access-dl9kv\") pod \"maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz\" (UID: \"75a852d2-c232-4e68-a79a-e945930aef35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz"
Nov 02 13:15:00.371908 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.371810    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a852d2-c232-4e68-a79a-e945930aef35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz\" (UID: \"75a852d2-c232-4e68-a79a-e945930aef35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz"
Nov 02 13:15:00.371908 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.371842    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71eddf4-e654-4f1b-817e-b6eab12f8571-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893755-67pcc\" (UID: \"e71eddf4-e654-4f1b-817e-b6eab12f8571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc"
Nov 02 13:15:00.372475 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.372456    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a852d2-c232-4e68-a79a-e945930aef35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz\" (UID: \"75a852d2-c232-4e68-a79a-e945930aef35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz"
Nov 02 13:15:00.372524 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.372491    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71eddf4-e654-4f1b-817e-b6eab12f8571-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893755-67pcc\" (UID: \"e71eddf4-e654-4f1b-817e-b6eab12f8571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc"
Nov 02 13:15:00.433722 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.433658    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf272\" (UniqueName: \"kubernetes.io/projected/e71eddf4-e654-4f1b-817e-b6eab12f8571-kube-api-access-vf272\") pod \"maas-api-key-cleanup-29893755-67pcc\" (UID: \"e71eddf4-e654-4f1b-817e-b6eab12f8571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc"
Nov 02 13:15:00.433722 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.433677    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl9kv\" (UniqueName: \"kubernetes.io/projected/75a852d2-c232-4e68-a79a-e945930aef35-kube-api-access-dl9kv\") pod \"maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz\" (UID: \"75a852d2-c232-4e68-a79a-e945930aef35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz"
Nov 02 13:15:00.495766 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.495737    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz"
Nov 02 13:15:00.501844 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.501817    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc"
Nov 02 13:15:00.540320 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.540288    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz"]
Nov 02 13:15:00.873065 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.873035    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz"]
Nov 02 13:15:00.874027 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:15:00.873990    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a852d2_c232_4e68_a79a_e945930aef35.slice/crio-38e675c6860dcf8f4fdd989a16a7533ef4861d43a68f74e4a36dca95d5db79bc WatchSource:0}: Error finding container 38e675c6860dcf8f4fdd989a16a7533ef4861d43a68f74e4a36dca95d5db79bc: Status 404 returned error can't find the container with id 38e675c6860dcf8f4fdd989a16a7533ef4861d43a68f74e4a36dca95d5db79bc
Nov 02 13:15:00.922575 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:00.922550    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc"]
Nov 02 13:15:00.923521 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:15:00.923497    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71eddf4_e654_4f1b_817e_b6eab12f8571.slice/crio-2a63dce350b74e3d2f0ee1fe00f590c0dac19cfc08f94b6fb3829a3096aea772 WatchSource:0}: Error finding container 2a63dce350b74e3d2f0ee1fe00f590c0dac19cfc08f94b6fb3829a3096aea772: Status 404 returned error can't find the container with id 2a63dce350b74e3d2f0ee1fe00f590c0dac19cfc08f94b6fb3829a3096aea772
Nov 02 13:15:01.034654 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:01.034610    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc" event={"ID":"e71eddf4-e654-4f1b-817e-b6eab12f8571","Type":"ContainerStarted","Data":"2a63dce350b74e3d2f0ee1fe00f590c0dac19cfc08f94b6fb3829a3096aea772"}
Nov 02 13:15:01.035777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:01.035746    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz" event={"ID":"75a852d2-c232-4e68-a79a-e945930aef35","Type":"ContainerStarted","Data":"38e675c6860dcf8f4fdd989a16a7533ef4861d43a68f74e4a36dca95d5db79bc"}
Nov 02 13:15:02.042455 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:02.042424    4464 generic.go:356] "Generic (PLEG): container finished" podID="e71eddf4-e654-4f1b-817e-b6eab12f8571" containerID="ebff7dbd34d35c1e72ff552fcd04cb62c918eafa97dc82ff6f6bf66b7748ae3c" exitCode=60
Nov 02 13:15:02.042909 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:02.042513    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc" event={"ID":"e71eddf4-e654-4f1b-817e-b6eab12f8571","Type":"ContainerDied","Data":"ebff7dbd34d35c1e72ff552fcd04cb62c918eafa97dc82ff6f6bf66b7748ae3c"}
Nov 02 13:15:02.042909 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:02.042671    4464 scope.go:122] "RemoveContainer" containerID="ebff7dbd34d35c1e72ff552fcd04cb62c918eafa97dc82ff6f6bf66b7748ae3c"
Nov 02 13:15:02.044200 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:02.044178    4464 generic.go:356] "Generic (PLEG): container finished" podID="75a852d2-c232-4e68-a79a-e945930aef35" containerID="90a97d7546887246ed797224f9a6d7848346d6825ee32d0130309abca5c00b29" exitCode=6
Nov 02 13:15:02.044315 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:02.044203    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz" event={"ID":"75a852d2-c232-4e68-a79a-e945930aef35","Type":"ContainerDied","Data":"90a97d7546887246ed797224f9a6d7848346d6825ee32d0130309abca5c00b29"}
Nov 02 13:15:02.171857 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:02.171832    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz"
Nov 02 13:15:02.290896 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:02.290863    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a852d2-c232-4e68-a79a-e945930aef35-kube-api-access-dl9kv\" (UniqueName: \"kubernetes.io/projected/75a852d2-c232-4e68-a79a-e945930aef35-kube-api-access-dl9kv\") pod \"75a852d2-c232-4e68-a79a-e945930aef35\" (UID: \"75a852d2-c232-4e68-a79a-e945930aef35\") "
Nov 02 13:15:02.291055 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:02.291010    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a852d2-c232-4e68-a79a-e945930aef35-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a852d2-c232-4e68-a79a-e945930aef35-maas-api-service-ca\") pod \"75a852d2-c232-4e68-a79a-e945930aef35\" (UID: \"75a852d2-c232-4e68-a79a-e945930aef35\") "
Nov 02 13:15:02.291327 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:02.291306    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a852d2-c232-4e68-a79a-e945930aef35-maas-api-service-ca" pod "75a852d2-c232-4e68-a79a-e945930aef35" (UID: "75a852d2-c232-4e68-a79a-e945930aef35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:02.292916 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:02.292876    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a852d2-c232-4e68-a79a-e945930aef35-kube-api-access-dl9kv" pod "75a852d2-c232-4e68-a79a-e945930aef35" (UID: "75a852d2-c232-4e68-a79a-e945930aef35"). InnerVolumeSpecName "kube-api-access-dl9kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:02.396506 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:02.393934    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a852d2-c232-4e68-a79a-e945930aef35-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:02.396506 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:02.393992    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl9kv\" (UniqueName: \"kubernetes.io/projected/75a852d2-c232-4e68-a79a-e945930aef35-kube-api-access-dl9kv\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:02.566327 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:02.566254    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"]
Nov 02 13:15:02.569180 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:02.569160    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c1934578-p7zg7"]
Nov 02 13:15:03.051304 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:03.051269    4464 generic.go:356] "Generic (PLEG): container finished" podID="e71eddf4-e654-4f1b-817e-b6eab12f8571" containerID="1142c9b707ddcbf6e669313da62caa343a4ac5532f61e937ae74f49c29be9a9d" exitCode=60
Nov 02 13:15:03.051785 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:03.051369    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc" event={"ID":"e71eddf4-e654-4f1b-817e-b6eab12f8571","Type":"ContainerDied","Data":"1142c9b707ddcbf6e669313da62caa343a4ac5532f61e937ae74f49c29be9a9d"}
Nov 02 13:15:03.051785 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:03.051428    4464 scope.go:122] "RemoveContainer" containerID="ebff7dbd34d35c1e72ff552fcd04cb62c918eafa97dc82ff6f6bf66b7748ae3c"
Nov 02 13:15:03.051785 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:03.051565    4464 scope.go:122] "RemoveContainer" containerID="1142c9b707ddcbf6e669313da62caa343a4ac5532f61e937ae74f49c29be9a9d"
Nov 02 13:15:03.051936 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:15:03.051814    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893755-67pcc_odh-ai-gateway-infra(e71eddf4-e654-4f1b-817e-b6eab12f8571)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc" podUID="e71eddf4-e654-4f1b-817e-b6eab12f8571"
Nov 02 13:15:03.052781 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:03.052760    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz"
Nov 02 13:15:03.052862 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:03.052766    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz" event={"ID":"75a852d2-c232-4e68-a79a-e945930aef35","Type":"ContainerDied","Data":"38e675c6860dcf8f4fdd989a16a7533ef4861d43a68f74e4a36dca95d5db79bc"}
Nov 02 13:15:03.063227 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:03.063216    4464 scope.go:122] "RemoveContainer" containerID="90a97d7546887246ed797224f9a6d7848346d6825ee32d0130309abca5c00b29"
Nov 02 13:15:03.313410 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:03.313378    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz"]
Nov 02 13:15:03.317130 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:03.317106    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-c1934578-29893755-pf4fz"]
Nov 02 13:15:03.582175 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:03.582086    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a852d2-c232-4e68-a79a-e945930aef35" path="/var/lib/kubelet/pods/75a852d2-c232-4e68-a79a-e945930aef35/volumes"
Nov 02 13:15:03.582593 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:03.582571    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf4a9a7-8d0d-4612-ab59-e0281234fa62" path="/var/lib/kubelet/pods/8bf4a9a7-8d0d-4612-ab59-e0281234fa62/volumes"
Nov 02 13:15:04.060139 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:04.060115    4464 scope.go:122] "RemoveContainer" containerID="1142c9b707ddcbf6e669313da62caa343a4ac5532f61e937ae74f49c29be9a9d"
Nov 02 13:15:04.060693 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:15:04.060325    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893755-67pcc_odh-ai-gateway-infra(e71eddf4-e654-4f1b-817e-b6eab12f8571)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc" podUID="e71eddf4-e654-4f1b-817e-b6eab12f8571"
Nov 02 13:15:04.999709 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:04.999674    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:15:07.394536 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:07.394498    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"]
Nov 02 13:15:07.395023 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:07.394763    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4" podUID="3c567554-60bf-46e3-8c4a-0a443cbeee13" containerName="istio-proxy" containerID="cri-o://9e2e2af4a8cc28f659ecfa2a23aabdf711a6ab0a9ec6929620ee0d0791012d76" gracePeriod=30
Nov 02 13:15:11.771294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:11.771258    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5975b94bc-pp42k"]
Nov 02 13:15:11.771683 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:11.771452    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5975b94bc-pp42k" podUID="3e629e5b-98b5-4987-83a2-dc5c8dfcb874" containerName="manager" containerID="cri-o://c100f80b15aa3a0a70d574ff1134fe4bb1ae76f4d2439675cc223144cc09556f" gracePeriod=30
Nov 02 13:15:12.016281 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.016258    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 13:15:12.081105 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.081036    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-kube-api-access-5hklq\" (UniqueName: \"kubernetes.io/projected/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-kube-api-access-5hklq\") pod \"3e629e5b-98b5-4987-83a2-dc5c8dfcb874\" (UID: \"3e629e5b-98b5-4987-83a2-dc5c8dfcb874\") "
Nov 02 13:15:12.081260 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.081127    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-webhook-cert\") pod \"3e629e5b-98b5-4987-83a2-dc5c8dfcb874\" (UID: \"3e629e5b-98b5-4987-83a2-dc5c8dfcb874\") "
Nov 02 13:15:12.081260 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.081155    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-metrics-certs\") pod \"3e629e5b-98b5-4987-83a2-dc5c8dfcb874\" (UID: \"3e629e5b-98b5-4987-83a2-dc5c8dfcb874\") "
Nov 02 13:15:12.083105 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.083077    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-kube-api-access-5hklq" pod "3e629e5b-98b5-4987-83a2-dc5c8dfcb874" (UID: "3e629e5b-98b5-4987-83a2-dc5c8dfcb874"). InnerVolumeSpecName "kube-api-access-5hklq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:12.083221 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.083189    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-metrics-certs" pod "3e629e5b-98b5-4987-83a2-dc5c8dfcb874" (UID: "3e629e5b-98b5-4987-83a2-dc5c8dfcb874"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:15:12.083260 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.083215    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-webhook-cert" pod "3e629e5b-98b5-4987-83a2-dc5c8dfcb874" (UID: "3e629e5b-98b5-4987-83a2-dc5c8dfcb874"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:15:12.100859 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.100834    4464 generic.go:356] "Generic (PLEG): container finished" podID="3e629e5b-98b5-4987-83a2-dc5c8dfcb874" containerID="c100f80b15aa3a0a70d574ff1134fe4bb1ae76f4d2439675cc223144cc09556f" exitCode=0
Nov 02 13:15:12.101002 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.100897    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5975b94bc-pp42k"
Nov 02 13:15:12.101002 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.100916    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5975b94bc-pp42k" event={"ID":"3e629e5b-98b5-4987-83a2-dc5c8dfcb874","Type":"ContainerDied","Data":"c100f80b15aa3a0a70d574ff1134fe4bb1ae76f4d2439675cc223144cc09556f"}
Nov 02 13:15:12.101002 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.100998    4464 scope.go:122] "RemoveContainer" containerID="c100f80b15aa3a0a70d574ff1134fe4bb1ae76f4d2439675cc223144cc09556f"
Nov 02 13:15:12.101181 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.100997    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5975b94bc-pp42k" event={"ID":"3e629e5b-98b5-4987-83a2-dc5c8dfcb874","Type":"ContainerDied","Data":"052c87f4d2dae13cf5ed7829e6a85ff6177bcec0bd2dfaf67b80edb53e5ec493"}
Nov 02 13:15:12.110737 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.110718    4464 scope.go:122] "RemoveContainer" containerID="2d2e9028093f07bfc6079e01c4b052b357fe115908a7f51f18bcb3c9f5422706"
Nov 02 13:15:12.118769 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.118754    4464 scope.go:122] "RemoveContainer" containerID="c100f80b15aa3a0a70d574ff1134fe4bb1ae76f4d2439675cc223144cc09556f"
Nov 02 13:15:12.119033 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:15:12.119013    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c100f80b15aa3a0a70d574ff1134fe4bb1ae76f4d2439675cc223144cc09556f\": container with ID starting with c100f80b15aa3a0a70d574ff1134fe4bb1ae76f4d2439675cc223144cc09556f not found: ID does not exist" containerID="c100f80b15aa3a0a70d574ff1134fe4bb1ae76f4d2439675cc223144cc09556f"
Nov 02 13:15:12.119093 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.119041    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c100f80b15aa3a0a70d574ff1134fe4bb1ae76f4d2439675cc223144cc09556f"} err="failed to get container status \"c100f80b15aa3a0a70d574ff1134fe4bb1ae76f4d2439675cc223144cc09556f\": rpc error: code = NotFound desc = could not find container \"c100f80b15aa3a0a70d574ff1134fe4bb1ae76f4d2439675cc223144cc09556f\": container with ID starting with c100f80b15aa3a0a70d574ff1134fe4bb1ae76f4d2439675cc223144cc09556f not found: ID does not exist"
Nov 02 13:15:12.119093 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.119058    4464 scope.go:122] "RemoveContainer" containerID="2d2e9028093f07bfc6079e01c4b052b357fe115908a7f51f18bcb3c9f5422706"
Nov 02 13:15:12.119318 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:15:12.119298    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2e9028093f07bfc6079e01c4b052b357fe115908a7f51f18bcb3c9f5422706\": container with ID starting with 2d2e9028093f07bfc6079e01c4b052b357fe115908a7f51f18bcb3c9f5422706 not found: ID does not exist" containerID="2d2e9028093f07bfc6079e01c4b052b357fe115908a7f51f18bcb3c9f5422706"
Nov 02 13:15:12.119380 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.119328    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2e9028093f07bfc6079e01c4b052b357fe115908a7f51f18bcb3c9f5422706"} err="failed to get container status \"2d2e9028093f07bfc6079e01c4b052b357fe115908a7f51f18bcb3c9f5422706\": rpc error: code = NotFound desc = could not find container \"2d2e9028093f07bfc6079e01c4b052b357fe115908a7f51f18bcb3c9f5422706\": container with ID starting with 2d2e9028093f07bfc6079e01c4b052b357fe115908a7f51f18bcb3c9f5422706 not found: ID does not exist"
Nov 02 13:15:12.182456 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.182428    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hklq\" (UniqueName: \"kubernetes.io/projected/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-kube-api-access-5hklq\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:12.182456 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.182454    4464 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-webhook-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:12.182670 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.182469    4464 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e629e5b-98b5-4987-83a2-dc5c8dfcb874-metrics-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:12.249996 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.249951    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5975b94bc-pp42k"]
Nov 02 13:15:12.254102 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.254070    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5975b94bc-pp42k"]
Nov 02 13:15:12.624745 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.624720    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:15:12.687613 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.687526    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-data\") pod \"3c567554-60bf-46e3-8c4a-0a443cbeee13\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") "
Nov 02 13:15:12.687613 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.687571    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c567554-60bf-46e3-8c4a-0a443cbeee13-kube-api-access-tg72v\" (UniqueName: \"kubernetes.io/projected/3c567554-60bf-46e3-8c4a-0a443cbeee13-kube-api-access-tg72v\") pod \"3c567554-60bf-46e3-8c4a-0a443cbeee13\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") "
Nov 02 13:15:12.687613 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.687589    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-token\" (UniqueName: \"kubernetes.io/projected/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-token\") pod \"3c567554-60bf-46e3-8c4a-0a443cbeee13\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") "
Nov 02 13:15:12.687918 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.687636    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-envoy\") pod \"3c567554-60bf-46e3-8c4a-0a443cbeee13\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") "
Nov 02 13:15:12.687918 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.687674    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-workload-certs\") pod \"3c567554-60bf-46e3-8c4a-0a443cbeee13\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") "
Nov 02 13:15:12.687918 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.687702    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-workload-socket\") pod \"3c567554-60bf-46e3-8c4a-0a443cbeee13\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") "
Nov 02 13:15:12.687918 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.687739    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c567554-60bf-46e3-8c4a-0a443cbeee13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c567554-60bf-46e3-8c4a-0a443cbeee13-istiod-ca-cert\") pod \"3c567554-60bf-46e3-8c4a-0a443cbeee13\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") "
Nov 02 13:15:12.687918 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.687762    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-credential-socket\") pod \"3c567554-60bf-46e3-8c4a-0a443cbeee13\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") "
Nov 02 13:15:12.687918 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.687805    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-podinfo\") pod \"3c567554-60bf-46e3-8c4a-0a443cbeee13\" (UID: \"3c567554-60bf-46e3-8c4a-0a443cbeee13\") "
Nov 02 13:15:12.687918 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.687860    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-data" pod "3c567554-60bf-46e3-8c4a-0a443cbeee13" (UID: "3c567554-60bf-46e3-8c4a-0a443cbeee13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:15:12.688323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.688076    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-workload-certs" pod "3c567554-60bf-46e3-8c4a-0a443cbeee13" (UID: "3c567554-60bf-46e3-8c4a-0a443cbeee13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:15:12.688323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.688147    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:12.688323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.688194    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-credential-socket" pod "3c567554-60bf-46e3-8c4a-0a443cbeee13" (UID: "3c567554-60bf-46e3-8c4a-0a443cbeee13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:15:12.688323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.688237    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c567554-60bf-46e3-8c4a-0a443cbeee13-istiod-ca-cert" pod "3c567554-60bf-46e3-8c4a-0a443cbeee13" (UID: "3c567554-60bf-46e3-8c4a-0a443cbeee13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:12.688323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.688252    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-workload-socket" pod "3c567554-60bf-46e3-8c4a-0a443cbeee13" (UID: "3c567554-60bf-46e3-8c4a-0a443cbeee13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:15:12.689902 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.689874    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-token" pod "3c567554-60bf-46e3-8c4a-0a443cbeee13" (UID: "3c567554-60bf-46e3-8c4a-0a443cbeee13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:12.690041 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.689920    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-envoy" pod "3c567554-60bf-46e3-8c4a-0a443cbeee13" (UID: "3c567554-60bf-46e3-8c4a-0a443cbeee13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:15:12.690041 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.689954    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c567554-60bf-46e3-8c4a-0a443cbeee13-kube-api-access-tg72v" pod "3c567554-60bf-46e3-8c4a-0a443cbeee13" (UID: "3c567554-60bf-46e3-8c4a-0a443cbeee13"). InnerVolumeSpecName "kube-api-access-tg72v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:12.690041 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.689962    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-podinfo" pod "3c567554-60bf-46e3-8c4a-0a443cbeee13" (UID: "3c567554-60bf-46e3-8c4a-0a443cbeee13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:15:12.788785 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.788746    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg72v\" (UniqueName: \"kubernetes.io/projected/3c567554-60bf-46e3-8c4a-0a443cbeee13-kube-api-access-tg72v\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:12.788785 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.788786    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:12.789294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.788803    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:12.789294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.788817    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:12.789294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.788831    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:12.789294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.788844    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c567554-60bf-46e3-8c4a-0a443cbeee13-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:12.789294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.788857    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c567554-60bf-46e3-8c4a-0a443cbeee13-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:12.789294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:12.788870    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c567554-60bf-46e3-8c4a-0a443cbeee13-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:13.066381 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.066350    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5975b94bc-2j9j5"]
Nov 02 13:15:13.066812 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.066798    4464 cpu_manager.go:414] "containerMap: removing container" podUID="3c567554-60bf-46e3-8c4a-0a443cbeee13" containerName="istio-proxy"
Nov 02 13:15:13.066871 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.066814    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c567554-60bf-46e3-8c4a-0a443cbeee13" containerName="istio-proxy"
Nov 02 13:15:13.066871 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.066832    4464 cpu_manager.go:414] "containerMap: removing container" podUID="75a852d2-c232-4e68-a79a-e945930aef35" containerName="cleanup"
Nov 02 13:15:13.066871 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.066837    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a852d2-c232-4e68-a79a-e945930aef35" containerName="cleanup"
Nov 02 13:15:13.066871 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.066857    4464 cpu_manager.go:414] "containerMap: removing container" podUID="3e629e5b-98b5-4987-83a2-dc5c8dfcb874" containerName="manager"
Nov 02 13:15:13.066871 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.066863    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e629e5b-98b5-4987-83a2-dc5c8dfcb874" containerName="manager"
Nov 02 13:15:13.067072 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.066874    4464 cpu_manager.go:414] "containerMap: removing container" podUID="3e629e5b-98b5-4987-83a2-dc5c8dfcb874" containerName="manager"
Nov 02 13:15:13.067072 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.066881    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e629e5b-98b5-4987-83a2-dc5c8dfcb874" containerName="manager"
Nov 02 13:15:13.067072 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.066946    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c567554-60bf-46e3-8c4a-0a443cbeee13" containerName="istio-proxy"
Nov 02 13:15:13.067072 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.066959    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e629e5b-98b5-4987-83a2-dc5c8dfcb874" containerName="manager"
Nov 02 13:15:13.067072 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.066970    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a852d2-c232-4e68-a79a-e945930aef35" containerName="cleanup"
Nov 02 13:15:13.070282 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.070265    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5975b94bc-2j9j5"
Nov 02 13:15:13.091887 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.091863    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hg6z\" (UniqueName: \"kubernetes.io/projected/f4503bbb-283e-431a-beb8-250a18584206-kube-api-access-5hg6z\") pod \"maas-controller-5975b94bc-2j9j5\" (UID: \"f4503bbb-283e-431a-beb8-250a18584206\") " pod="opendatahub/maas-controller-5975b94bc-2j9j5"
Nov 02 13:15:13.092027 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.091907    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f4503bbb-283e-431a-beb8-250a18584206-webhook-cert\") pod \"maas-controller-5975b94bc-2j9j5\" (UID: \"f4503bbb-283e-431a-beb8-250a18584206\") " pod="opendatahub/maas-controller-5975b94bc-2j9j5"
Nov 02 13:15:13.092027 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.091933    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4503bbb-283e-431a-beb8-250a18584206-metrics-certs\") pod \"maas-controller-5975b94bc-2j9j5\" (UID: \"f4503bbb-283e-431a-beb8-250a18584206\") " pod="opendatahub/maas-controller-5975b94bc-2j9j5"
Nov 02 13:15:13.107025 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.106997    4464 generic.go:356] "Generic (PLEG): container finished" podID="3c567554-60bf-46e3-8c4a-0a443cbeee13" containerID="9e2e2af4a8cc28f659ecfa2a23aabdf711a6ab0a9ec6929620ee0d0791012d76" exitCode=0
Nov 02 13:15:13.107137 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.107025    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4" event={"ID":"3c567554-60bf-46e3-8c4a-0a443cbeee13","Type":"ContainerDied","Data":"9e2e2af4a8cc28f659ecfa2a23aabdf711a6ab0a9ec6929620ee0d0791012d76"}
Nov 02 13:15:13.107137 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.107054    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"
Nov 02 13:15:13.107137 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.107079    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4" event={"ID":"3c567554-60bf-46e3-8c4a-0a443cbeee13","Type":"ContainerDied","Data":"64047019ad213f87bd9050ae60750de775f4e14fb0ec680d27f5840e41f15111"}
Nov 02 13:15:13.107137 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.107104    4464 scope.go:122] "RemoveContainer" containerID="9e2e2af4a8cc28f659ecfa2a23aabdf711a6ab0a9ec6929620ee0d0791012d76"
Nov 02 13:15:13.121180 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.121130    4464 scope.go:122] "RemoveContainer" containerID="a3458d8cbe842951a119657f1975d2c8f70ca644ac1697861afd0754a852b15c"
Nov 02 13:15:13.123955 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.123936    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 13:15:13.124071 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.123991    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 13:15:13.124071 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.124021    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tzzlq\""
Nov 02 13:15:13.126542 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.126502    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 13:15:13.129340 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.129319    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5975b94bc-2j9j5"]
Nov 02 13:15:13.132136 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.132114    4464 scope.go:122] "RemoveContainer" containerID="9e2e2af4a8cc28f659ecfa2a23aabdf711a6ab0a9ec6929620ee0d0791012d76"
Nov 02 13:15:13.132479 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:15:13.132453    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e2e2af4a8cc28f659ecfa2a23aabdf711a6ab0a9ec6929620ee0d0791012d76\": container with ID starting with 9e2e2af4a8cc28f659ecfa2a23aabdf711a6ab0a9ec6929620ee0d0791012d76 not found: ID does not exist" containerID="9e2e2af4a8cc28f659ecfa2a23aabdf711a6ab0a9ec6929620ee0d0791012d76"
Nov 02 13:15:13.132552 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.132490    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e2e2af4a8cc28f659ecfa2a23aabdf711a6ab0a9ec6929620ee0d0791012d76"} err="failed to get container status \"9e2e2af4a8cc28f659ecfa2a23aabdf711a6ab0a9ec6929620ee0d0791012d76\": rpc error: code = NotFound desc = could not find container \"9e2e2af4a8cc28f659ecfa2a23aabdf711a6ab0a9ec6929620ee0d0791012d76\": container with ID starting with 9e2e2af4a8cc28f659ecfa2a23aabdf711a6ab0a9ec6929620ee0d0791012d76 not found: ID does not exist"
Nov 02 13:15:13.132552 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.132512    4464 scope.go:122] "RemoveContainer" containerID="a3458d8cbe842951a119657f1975d2c8f70ca644ac1697861afd0754a852b15c"
Nov 02 13:15:13.132802 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:15:13.132784    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3458d8cbe842951a119657f1975d2c8f70ca644ac1697861afd0754a852b15c\": container with ID starting with a3458d8cbe842951a119657f1975d2c8f70ca644ac1697861afd0754a852b15c not found: ID does not exist" containerID="a3458d8cbe842951a119657f1975d2c8f70ca644ac1697861afd0754a852b15c"
Nov 02 13:15:13.132865 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.132807    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3458d8cbe842951a119657f1975d2c8f70ca644ac1697861afd0754a852b15c"} err="failed to get container status \"a3458d8cbe842951a119657f1975d2c8f70ca644ac1697861afd0754a852b15c\": rpc error: code = NotFound desc = could not find container \"a3458d8cbe842951a119657f1975d2c8f70ca644ac1697861afd0754a852b15c\": container with ID starting with a3458d8cbe842951a119657f1975d2c8f70ca644ac1697861afd0754a852b15c not found: ID does not exist"
Nov 02 13:15:13.192900 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.192872    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hg6z\" (UniqueName: \"kubernetes.io/projected/f4503bbb-283e-431a-beb8-250a18584206-kube-api-access-5hg6z\") pod \"maas-controller-5975b94bc-2j9j5\" (UID: \"f4503bbb-283e-431a-beb8-250a18584206\") " pod="opendatahub/maas-controller-5975b94bc-2j9j5"
Nov 02 13:15:13.193026 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.192920    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f4503bbb-283e-431a-beb8-250a18584206-webhook-cert\") pod \"maas-controller-5975b94bc-2j9j5\" (UID: \"f4503bbb-283e-431a-beb8-250a18584206\") " pod="opendatahub/maas-controller-5975b94bc-2j9j5"
Nov 02 13:15:13.193026 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.192998    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4503bbb-283e-431a-beb8-250a18584206-metrics-certs\") pod \"maas-controller-5975b94bc-2j9j5\" (UID: \"f4503bbb-283e-431a-beb8-250a18584206\") " pod="opendatahub/maas-controller-5975b94bc-2j9j5"
Nov 02 13:15:13.195372 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.195346    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f4503bbb-283e-431a-beb8-250a18584206-webhook-cert\") pod \"maas-controller-5975b94bc-2j9j5\" (UID: \"f4503bbb-283e-431a-beb8-250a18584206\") " pod="opendatahub/maas-controller-5975b94bc-2j9j5"
Nov 02 13:15:13.195481 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.195375    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4503bbb-283e-431a-beb8-250a18584206-metrics-certs\") pod \"maas-controller-5975b94bc-2j9j5\" (UID: \"f4503bbb-283e-431a-beb8-250a18584206\") " pod="opendatahub/maas-controller-5975b94bc-2j9j5"
Nov 02 13:15:13.254310 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.254288    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hg6z\" (UniqueName: \"kubernetes.io/projected/f4503bbb-283e-431a-beb8-250a18584206-kube-api-access-5hg6z\") pod \"maas-controller-5975b94bc-2j9j5\" (UID: \"f4503bbb-283e-431a-beb8-250a18584206\") " pod="opendatahub/maas-controller-5975b94bc-2j9j5"
Nov 02 13:15:13.303373 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.303345    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"]
Nov 02 13:15:13.307268 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.307241    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c1934578-openshift-default-9fd49846f-98hx4"]
Nov 02 13:15:13.382747 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.382670    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5975b94bc-2j9j5"
Nov 02 13:15:13.580691 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.580659    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c567554-60bf-46e3-8c4a-0a443cbeee13" path="/var/lib/kubelet/pods/3c567554-60bf-46e3-8c4a-0a443cbeee13/volumes"
Nov 02 13:15:13.581056 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.581043    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e629e5b-98b5-4987-83a2-dc5c8dfcb874" path="/var/lib/kubelet/pods/3e629e5b-98b5-4987-83a2-dc5c8dfcb874/volumes"
Nov 02 13:15:13.761718 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.761693    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5975b94bc-2j9j5"]
Nov 02 13:15:13.762951 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:15:13.762929    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4503bbb_283e_431a_beb8_250a18584206.slice/crio-df46001fc93b4dea87e31c93d4c413d4e93bcc36fcb5144094d9c58d190a51c8 WatchSource:0}: Error finding container df46001fc93b4dea87e31c93d4c413d4e93bcc36fcb5144094d9c58d190a51c8: Status 404 returned error can't find the container with id df46001fc93b4dea87e31c93d4c413d4e93bcc36fcb5144094d9c58d190a51c8
Nov 02 13:15:13.764358 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:13.764344    4464 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:15:14.114718 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:14.114634    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5975b94bc-2j9j5" event={"ID":"f4503bbb-283e-431a-beb8-250a18584206","Type":"ContainerStarted","Data":"df46001fc93b4dea87e31c93d4c413d4e93bcc36fcb5144094d9c58d190a51c8"}
Nov 02 13:15:14.574701 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:14.574684    4464 scope.go:122] "RemoveContainer" containerID="1142c9b707ddcbf6e669313da62caa343a4ac5532f61e937ae74f49c29be9a9d"
Nov 02 13:15:15.120911 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:15.120821    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5975b94bc-2j9j5" event={"ID":"f4503bbb-283e-431a-beb8-250a18584206","Type":"ContainerStarted","Data":"c0b24842bef74fbcdb7196d6a72cb94e703666927a4470bf78f90f8f406452ab"}
Nov 02 13:15:15.120911 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:15.120891    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5975b94bc-2j9j5"
Nov 02 13:15:15.122812 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:15.122788    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc" event={"ID":"e71eddf4-e654-4f1b-817e-b6eab12f8571","Type":"ContainerStarted","Data":"e9aaa6464b6f0d7a30004ae4eb537ece82b3c03cc5de6294eb1a72b293fbe276"}
Nov 02 13:15:15.244227 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:15.244166    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5975b94bc-2j9j5" podStartSLOduration=1.339855088 podStartE2EDuration="2.244152019s" totalImagesPullingTime="904.296931ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:15.242194386 +0000 UTC m=+4743.805902796" watchObservedRunningTime="2026-11-02 13:15:15.244152019 +0000 UTC m=+4743.807860421"
Nov 02 13:15:15.362852 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:15.362789    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc" podStartSLOduration=15.3627689 podStartE2EDuration="15.3627689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:15.360225046 +0000 UTC m=+4743.923933454" watchObservedRunningTime="2026-11-02 13:15:15.3627689 +0000 UTC m=+4743.926477310"
Nov 02 13:15:16.129861 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:16.129830    4464 generic.go:356] "Generic (PLEG): container finished" podID="e71eddf4-e654-4f1b-817e-b6eab12f8571" containerID="e9aaa6464b6f0d7a30004ae4eb537ece82b3c03cc5de6294eb1a72b293fbe276" exitCode=60
Nov 02 13:15:16.130313 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:16.129909    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc" event={"ID":"e71eddf4-e654-4f1b-817e-b6eab12f8571","Type":"ContainerDied","Data":"e9aaa6464b6f0d7a30004ae4eb537ece82b3c03cc5de6294eb1a72b293fbe276"}
Nov 02 13:15:16.130313 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:16.129965    4464 scope.go:122] "RemoveContainer" containerID="1142c9b707ddcbf6e669313da62caa343a4ac5532f61e937ae74f49c29be9a9d"
Nov 02 13:15:16.130408 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:16.130393    4464 scope.go:122] "RemoveContainer" containerID="e9aaa6464b6f0d7a30004ae4eb537ece82b3c03cc5de6294eb1a72b293fbe276"
Nov 02 13:15:16.130572 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:15:16.130555    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893755-67pcc_odh-ai-gateway-infra(e71eddf4-e654-4f1b-817e-b6eab12f8571)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc" podUID="e71eddf4-e654-4f1b-817e-b6eab12f8571"
Nov 02 13:15:16.370611 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:16.370575    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc"]
Nov 02 13:15:17.262308 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:17.262285    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc"
Nov 02 13:15:17.328272 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:17.328247    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71eddf4-e654-4f1b-817e-b6eab12f8571-kube-api-access-vf272\" (UniqueName: \"kubernetes.io/projected/e71eddf4-e654-4f1b-817e-b6eab12f8571-kube-api-access-vf272\") pod \"e71eddf4-e654-4f1b-817e-b6eab12f8571\" (UID: \"e71eddf4-e654-4f1b-817e-b6eab12f8571\") "
Nov 02 13:15:17.328397 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:17.328343    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e71eddf4-e654-4f1b-817e-b6eab12f8571-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71eddf4-e654-4f1b-817e-b6eab12f8571-maas-api-service-ca\") pod \"e71eddf4-e654-4f1b-817e-b6eab12f8571\" (UID: \"e71eddf4-e654-4f1b-817e-b6eab12f8571\") "
Nov 02 13:15:17.328659 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:17.328639    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71eddf4-e654-4f1b-817e-b6eab12f8571-maas-api-service-ca" pod "e71eddf4-e654-4f1b-817e-b6eab12f8571" (UID: "e71eddf4-e654-4f1b-817e-b6eab12f8571"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:17.330413 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:17.330392    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71eddf4-e654-4f1b-817e-b6eab12f8571-kube-api-access-vf272" pod "e71eddf4-e654-4f1b-817e-b6eab12f8571" (UID: "e71eddf4-e654-4f1b-817e-b6eab12f8571"). InnerVolumeSpecName "kube-api-access-vf272". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:17.429672 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:17.429595    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf272\" (UniqueName: \"kubernetes.io/projected/e71eddf4-e654-4f1b-817e-b6eab12f8571-kube-api-access-vf272\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:17.429672 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:17.429632    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71eddf4-e654-4f1b-817e-b6eab12f8571-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:18.141560 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:18.141528    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc"
Nov 02 13:15:18.141715 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:18.141561    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc" event={"ID":"e71eddf4-e654-4f1b-817e-b6eab12f8571","Type":"ContainerDied","Data":"2a63dce350b74e3d2f0ee1fe00f590c0dac19cfc08f94b6fb3829a3096aea772"}
Nov 02 13:15:18.141715 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:18.141632    4464 scope.go:122] "RemoveContainer" containerID="e9aaa6464b6f0d7a30004ae4eb537ece82b3c03cc5de6294eb1a72b293fbe276"
Nov 02 13:15:18.309752 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:18.309714    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc"]
Nov 02 13:15:18.360355 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:18.360322    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893755-67pcc"]
Nov 02 13:15:19.581137 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:19.581105    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e71eddf4-e654-4f1b-817e-b6eab12f8571" path="/var/lib/kubelet/pods/e71eddf4-e654-4f1b-817e-b6eab12f8571/volumes"
Nov 02 13:15:26.132406 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:26.132374    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5975b94bc-2j9j5"
Nov 02 13:15:28.290968 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:15:28.290930    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:15:33.216585 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.216548    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"]
Nov 02 13:15:33.217186 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.217165    4464 cpu_manager.go:414] "containerMap: removing container" podUID="3c567554-60bf-46e3-8c4a-0a443cbeee13" containerName="istio-proxy"
Nov 02 13:15:33.217292 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.217187    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c567554-60bf-46e3-8c4a-0a443cbeee13" containerName="istio-proxy"
Nov 02 13:15:33.217292 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.217218    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e71eddf4-e654-4f1b-817e-b6eab12f8571" containerName="cleanup"
Nov 02 13:15:33.217292 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.217226    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71eddf4-e654-4f1b-817e-b6eab12f8571" containerName="cleanup"
Nov 02 13:15:33.217292 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.217258    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e71eddf4-e654-4f1b-817e-b6eab12f8571" containerName="cleanup"
Nov 02 13:15:33.217292 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.217268    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71eddf4-e654-4f1b-817e-b6eab12f8571" containerName="cleanup"
Nov 02 13:15:33.217292 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.217287    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e71eddf4-e654-4f1b-817e-b6eab12f8571" containerName="cleanup"
Nov 02 13:15:33.217624 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.217294    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71eddf4-e654-4f1b-817e-b6eab12f8571" containerName="cleanup"
Nov 02 13:15:33.217624 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.217397    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71eddf4-e654-4f1b-817e-b6eab12f8571" containerName="cleanup"
Nov 02 13:15:33.217624 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.217410    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71eddf4-e654-4f1b-817e-b6eab12f8571" containerName="cleanup"
Nov 02 13:15:33.217624 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.217422    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c567554-60bf-46e3-8c4a-0a443cbeee13" containerName="istio-proxy"
Nov 02 13:15:33.217624 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.217434    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e629e5b-98b5-4987-83a2-dc5c8dfcb874" containerName="manager"
Nov 02 13:15:33.217624 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.217444    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71eddf4-e654-4f1b-817e-b6eab12f8571" containerName="cleanup"
Nov 02 13:15:33.220591 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.220569    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"
Nov 02 13:15:33.269687 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.269660    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de67aac-8808-4f87-ac9a-83c55cda1397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-d8rqr\" (UID: \"4de67aac-8808-4f87-ac9a-83c55cda1397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"
Nov 02 13:15:33.269826 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.269694    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4z5j\" (UniqueName: \"kubernetes.io/projected/4de67aac-8808-4f87-ac9a-83c55cda1397-kube-api-access-b4z5j\") pod \"maas-api-revoke-sub-85237e45-d8rqr\" (UID: \"4de67aac-8808-4f87-ac9a-83c55cda1397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"
Nov 02 13:15:33.269826 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.269724    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de67aac-8808-4f87-ac9a-83c55cda1397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-d8rqr\" (UID: \"4de67aac-8808-4f87-ac9a-83c55cda1397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"
Nov 02 13:15:33.275672 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.275648    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:15:33.275825 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.275672    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:15:33.326304 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.326282    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:15:33.326568 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.326549    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:15:33.328646 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.328628    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"]
Nov 02 13:15:33.370925 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.370894    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de67aac-8808-4f87-ac9a-83c55cda1397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-d8rqr\" (UID: \"4de67aac-8808-4f87-ac9a-83c55cda1397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"
Nov 02 13:15:33.371091 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.370928    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4z5j\" (UniqueName: \"kubernetes.io/projected/4de67aac-8808-4f87-ac9a-83c55cda1397-kube-api-access-b4z5j\") pod \"maas-api-revoke-sub-85237e45-d8rqr\" (UID: \"4de67aac-8808-4f87-ac9a-83c55cda1397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"
Nov 02 13:15:33.371162 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.371142    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de67aac-8808-4f87-ac9a-83c55cda1397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-d8rqr\" (UID: \"4de67aac-8808-4f87-ac9a-83c55cda1397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"
Nov 02 13:15:33.371580 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.371556    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de67aac-8808-4f87-ac9a-83c55cda1397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-d8rqr\" (UID: \"4de67aac-8808-4f87-ac9a-83c55cda1397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"
Nov 02 13:15:33.373501 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.373480    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de67aac-8808-4f87-ac9a-83c55cda1397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-d8rqr\" (UID: \"4de67aac-8808-4f87-ac9a-83c55cda1397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"
Nov 02 13:15:33.432719 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.432694    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4z5j\" (UniqueName: \"kubernetes.io/projected/4de67aac-8808-4f87-ac9a-83c55cda1397-kube-api-access-b4z5j\") pod \"maas-api-revoke-sub-85237e45-d8rqr\" (UID: \"4de67aac-8808-4f87-ac9a-83c55cda1397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"
Nov 02 13:15:33.532587 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.532557    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"
Nov 02 13:15:33.903301 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:33.903275    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"]
Nov 02 13:15:33.904775 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:15:33.904747    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de67aac_8808_4f87_ac9a_83c55cda1397.slice/crio-caac140b8417d215b2c7c0838450ff78075272565f6bd68d5480534c76b024d6 WatchSource:0}: Error finding container caac140b8417d215b2c7c0838450ff78075272565f6bd68d5480534c76b024d6: Status 404 returned error can't find the container with id caac140b8417d215b2c7c0838450ff78075272565f6bd68d5480534c76b024d6
Nov 02 13:15:34.221040 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:34.221002    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr" event={"ID":"4de67aac-8808-4f87-ac9a-83c55cda1397","Type":"ContainerStarted","Data":"52f3ae959275858316282b82459f7537e2e185d655d3cfa3f2778cb9466f2920"}
Nov 02 13:15:34.221425 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:34.221057    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr" event={"ID":"4de67aac-8808-4f87-ac9a-83c55cda1397","Type":"ContainerStarted","Data":"caac140b8417d215b2c7c0838450ff78075272565f6bd68d5480534c76b024d6"}
Nov 02 13:15:34.340203 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:34.340156    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr" podStartSLOduration=1.340142371 podStartE2EDuration="1.340142371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:34.338249893 +0000 UTC m=+4762.901958297" watchObservedRunningTime="2026-11-02 13:15:34.340142371 +0000 UTC m=+4762.903850780"
Nov 02 13:15:35.227680 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:35.227646    4464 generic.go:356] "Generic (PLEG): container finished" podID="4de67aac-8808-4f87-ac9a-83c55cda1397" containerID="52f3ae959275858316282b82459f7537e2e185d655d3cfa3f2778cb9466f2920" exitCode=0
Nov 02 13:15:35.228147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:35.227734    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr" event={"ID":"4de67aac-8808-4f87-ac9a-83c55cda1397","Type":"ContainerDied","Data":"52f3ae959275858316282b82459f7537e2e185d655d3cfa3f2778cb9466f2920"}
Nov 02 13:15:36.359889 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:36.359864    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"
Nov 02 13:15:36.499713 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:36.499642    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4de67aac-8808-4f87-ac9a-83c55cda1397-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de67aac-8808-4f87-ac9a-83c55cda1397-maas-api-openshift-ca\") pod \"4de67aac-8808-4f87-ac9a-83c55cda1397\" (UID: \"4de67aac-8808-4f87-ac9a-83c55cda1397\") "
Nov 02 13:15:36.499713 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:36.499692    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4de67aac-8808-4f87-ac9a-83c55cda1397-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de67aac-8808-4f87-ac9a-83c55cda1397-maas-api-xks-ca\") pod \"4de67aac-8808-4f87-ac9a-83c55cda1397\" (UID: \"4de67aac-8808-4f87-ac9a-83c55cda1397\") "
Nov 02 13:15:36.499934 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:36.499794    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de67aac-8808-4f87-ac9a-83c55cda1397-kube-api-access-b4z5j\" (UniqueName: \"kubernetes.io/projected/4de67aac-8808-4f87-ac9a-83c55cda1397-kube-api-access-b4z5j\") pod \"4de67aac-8808-4f87-ac9a-83c55cda1397\" (UID: \"4de67aac-8808-4f87-ac9a-83c55cda1397\") "
Nov 02 13:15:36.500031 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:36.500013    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de67aac-8808-4f87-ac9a-83c55cda1397-maas-api-openshift-ca" pod "4de67aac-8808-4f87-ac9a-83c55cda1397" (UID: "4de67aac-8808-4f87-ac9a-83c55cda1397"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:36.501683 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:36.501661    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4de67aac-8808-4f87-ac9a-83c55cda1397-maas-api-xks-ca" pod "4de67aac-8808-4f87-ac9a-83c55cda1397" (UID: "4de67aac-8808-4f87-ac9a-83c55cda1397"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:15:36.501811 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:36.501780    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de67aac-8808-4f87-ac9a-83c55cda1397-kube-api-access-b4z5j" pod "4de67aac-8808-4f87-ac9a-83c55cda1397" (UID: "4de67aac-8808-4f87-ac9a-83c55cda1397"). InnerVolumeSpecName "kube-api-access-b4z5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:36.601471 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:36.601440    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4z5j\" (UniqueName: \"kubernetes.io/projected/4de67aac-8808-4f87-ac9a-83c55cda1397-kube-api-access-b4z5j\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:36.601471 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:36.601467    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de67aac-8808-4f87-ac9a-83c55cda1397-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:36.601661 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:36.601479    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de67aac-8808-4f87-ac9a-83c55cda1397-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:37.239618 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:37.239580    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr" event={"ID":"4de67aac-8808-4f87-ac9a-83c55cda1397","Type":"ContainerDied","Data":"caac140b8417d215b2c7c0838450ff78075272565f6bd68d5480534c76b024d6"}
Nov 02 13:15:37.239618 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:37.239614    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"
Nov 02 13:15:37.239820 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:37.239632    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caac140b8417d215b2c7c0838450ff78075272565f6bd68d5480534c76b024d6"
Nov 02 13:15:41.234601 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.234568    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"]
Nov 02 13:15:41.235012 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.234970    4464 cpu_manager.go:414] "containerMap: removing container" podUID="4de67aac-8808-4f87-ac9a-83c55cda1397" containerName="revoke-keys"
Nov 02 13:15:41.235012 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.234995    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de67aac-8808-4f87-ac9a-83c55cda1397" containerName="revoke-keys"
Nov 02 13:15:41.235095 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.235080    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de67aac-8808-4f87-ac9a-83c55cda1397" containerName="revoke-keys"
Nov 02 13:15:41.238358 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.238337    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.285348 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.285322    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"]
Nov 02 13:15:41.288683 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.288667    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.301048 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.301009    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c5ldd\""
Nov 02 13:15:41.339520 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.339492    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.339807 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.339525    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5780a48-9777-4e56-87b4-e0d00bebb615-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.339807 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.339553    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.339807 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.339615    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.339807 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.339746    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.340237 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.339826    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.340237 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.339872    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zsz5\" (UniqueName: \"kubernetes.io/projected/e5780a48-9777-4e56-87b4-e0d00bebb615-kube-api-access-4zsz5\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.340237 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.339940    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.340237 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.340019    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.343566 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.343545    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"]
Nov 02 13:15:41.402755 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.402715    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"]
Nov 02 13:15:41.441509 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.441480    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.441681 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.441514    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.441681 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.441534    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.441681 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.441560    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.441681 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.441633    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.441681 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.441668    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.441969 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.441684    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.441969 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.441705    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqr7m\" (UniqueName: \"kubernetes.io/projected/931c4f3d-1712-4c42-9226-ec630f19f7ca-kube-api-access-nqr7m\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.441969 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.441781    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zsz5\" (UniqueName: \"kubernetes.io/projected/e5780a48-9777-4e56-87b4-e0d00bebb615-kube-api-access-4zsz5\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.441969 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.441809    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.441969 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.441845    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.441969 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.441912    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931c4f3d-1712-4c42-9226-ec630f19f7ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.441969 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.441945    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.442391 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.441993    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.442391 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.442063    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.442391 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.442089    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.442391 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.442091    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.442391 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.442138    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.442391 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.442203    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.442391 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.442210    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.442391 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.442311    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5780a48-9777-4e56-87b4-e0d00bebb615-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.442391 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.442348    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.442870 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.442852    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5780a48-9777-4e56-87b4-e0d00bebb615-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.444533 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.444504    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.444675 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.444652    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.503275 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.503221    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.503417 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.503399    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zsz5\" (UniqueName: \"kubernetes.io/projected/e5780a48-9777-4e56-87b4-e0d00bebb615-kube-api-access-4zsz5\") pod \"maas-default-gateway-openshift-default-6dcf757899-ppqp4\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.543729 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.543704    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.543831 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.543765    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.543831 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.543793    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.543831 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.543812    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqr7m\" (UniqueName: \"kubernetes.io/projected/931c4f3d-1712-4c42-9226-ec630f19f7ca-kube-api-access-nqr7m\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.543946 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.543837    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.543946 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.543874    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931c4f3d-1712-4c42-9226-ec630f19f7ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.543946 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.543905    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.543946 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.543933    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.544180 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.543956    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.544180 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.544112    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.544180 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.544140    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.544348 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.544283    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.544690 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.544665    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.544774 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.544665    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931c4f3d-1712-4c42-9226-ec630f19f7ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.545698 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.545677    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.546381 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.546358    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.551083 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.551068    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:41.605964 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.605932    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqr7m\" (UniqueName: \"kubernetes.io/projected/931c4f3d-1712-4c42-9226-ec630f19f7ca-kube-api-access-nqr7m\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.606591 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.606568    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-7xbnw\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.900959 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.900881    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:41.974802 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.974776    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"]
Nov 02 13:15:41.975838 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:15:41.975811    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5780a48_9777_4e56_87b4_e0d00bebb615.slice/crio-abff9ec801027dbfdd460e7b853dc4e522936e319c0edb14eb555c68c72975bd WatchSource:0}: Error finding container abff9ec801027dbfdd460e7b853dc4e522936e319c0edb14eb555c68c72975bd: Status 404 returned error can't find the container with id abff9ec801027dbfdd460e7b853dc4e522936e319c0edb14eb555c68c72975bd
Nov 02 13:15:41.978138 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.978094    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:15:41.978228 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.978182    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:15:41.978228 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:41.978224    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:15:42.266370 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:42.266328    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4" event={"ID":"e5780a48-9777-4e56-87b4-e0d00bebb615","Type":"ContainerStarted","Data":"f636c4ad6cb07893d2e3492b3a3d49d16dc6e026a5f8f80d837b629e6f4f2ec8"}
Nov 02 13:15:42.266829 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:42.266389    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4" event={"ID":"e5780a48-9777-4e56-87b4-e0d00bebb615","Type":"ContainerStarted","Data":"abff9ec801027dbfdd460e7b853dc4e522936e319c0edb14eb555c68c72975bd"}
Nov 02 13:15:42.275665 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:42.275642    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"]
Nov 02 13:15:42.281023 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:15:42.280967    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931c4f3d_1712_4c42_9226_ec630f19f7ca.slice/crio-9a2b0bc04239fe0a9b08b0231d6a182cbf1b1fa5640cac1dd2cd34d6128ed97a WatchSource:0}: Error finding container 9a2b0bc04239fe0a9b08b0231d6a182cbf1b1fa5640cac1dd2cd34d6128ed97a: Status 404 returned error can't find the container with id 9a2b0bc04239fe0a9b08b0231d6a182cbf1b1fa5640cac1dd2cd34d6128ed97a
Nov 02 13:15:42.283381 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:42.283340    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:15:42.283511 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:42.283421    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:15:42.283511 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:42.283451    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:15:42.391457 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:42.391407    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4" podStartSLOduration=1.391387312 podStartE2EDuration="1.391387312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:42.388358337 +0000 UTC m=+4770.952066751" watchObservedRunningTime="2026-11-02 13:15:42.391387312 +0000 UTC m=+4770.955095722"
Nov 02 13:15:42.552102 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:42.552015    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:43.302625 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:43.302573    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw" event={"ID":"931c4f3d-1712-4c42-9226-ec630f19f7ca","Type":"ContainerStarted","Data":"cbf6f5e093692f19a40aa6688bb1759ccc1429d5259ece4df8c79a6211818aa2"}
Nov 02 13:15:43.303275 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:43.303246    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw" event={"ID":"931c4f3d-1712-4c42-9226-ec630f19f7ca","Type":"ContainerStarted","Data":"9a2b0bc04239fe0a9b08b0231d6a182cbf1b1fa5640cac1dd2cd34d6128ed97a"}
Nov 02 13:15:43.480998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:43.478892    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw" podStartSLOduration=2.47886994 podStartE2EDuration="2.47886994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:43.477257704 +0000 UTC m=+4772.040966114" watchObservedRunningTime="2026-11-02 13:15:43.47886994 +0000 UTC m=+4772.042578351"
Nov 02 13:15:43.553127 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:43.553038    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ppqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Nov 02 13:15:43.553127 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:43.553107    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4" podUID="e5780a48-9777-4e56-87b4-e0d00bebb615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Nov 02 13:15:43.902826 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:43.902741    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:43.903213 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:43.903190    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7xbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 02 13:15:43.903381 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:43.903356    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw" podUID="931c4f3d-1712-4c42-9226-ec630f19f7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 02 13:15:44.551784 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:44.551749    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ppqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Nov 02 13:15:44.552224 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:44.551803    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4" podUID="e5780a48-9777-4e56-87b4-e0d00bebb615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Nov 02 13:15:44.902057 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:44.901512    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7xbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 02 13:15:44.902057 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:44.901592    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw" podUID="931c4f3d-1712-4c42-9226-ec630f19f7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 02 13:15:45.551611 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:45.551580    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ppqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Nov 02 13:15:45.551802 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:45.551629    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4" podUID="e5780a48-9777-4e56-87b4-e0d00bebb615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Nov 02 13:15:45.739325 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:45.739288    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5975b94bc-2j9j5"]
Nov 02 13:15:45.739744 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:45.739499    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5975b94bc-2j9j5" podUID="f4503bbb-283e-431a-beb8-250a18584206" containerName="manager" containerID="cri-o://c0b24842bef74fbcdb7196d6a72cb94e703666927a4470bf78f90f8f406452ab" gracePeriod=30
Nov 02 13:15:45.913001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:45.911505    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7xbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 02 13:15:45.913001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:45.911566    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw" podUID="931c4f3d-1712-4c42-9226-ec630f19f7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 02 13:15:46.119541 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.119473    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5975b94bc-2j9j5"
Nov 02 13:15:46.138775 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.138753    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4503bbb-283e-431a-beb8-250a18584206-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4503bbb-283e-431a-beb8-250a18584206-metrics-certs\") pod \"f4503bbb-283e-431a-beb8-250a18584206\" (UID: \"f4503bbb-283e-431a-beb8-250a18584206\") "
Nov 02 13:15:46.138882 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.138839    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4503bbb-283e-431a-beb8-250a18584206-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f4503bbb-283e-431a-beb8-250a18584206-webhook-cert\") pod \"f4503bbb-283e-431a-beb8-250a18584206\" (UID: \"f4503bbb-283e-431a-beb8-250a18584206\") "
Nov 02 13:15:46.138945 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.138915    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4503bbb-283e-431a-beb8-250a18584206-kube-api-access-5hg6z\" (UniqueName: \"kubernetes.io/projected/f4503bbb-283e-431a-beb8-250a18584206-kube-api-access-5hg6z\") pod \"f4503bbb-283e-431a-beb8-250a18584206\" (UID: \"f4503bbb-283e-431a-beb8-250a18584206\") "
Nov 02 13:15:46.140770 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.140746    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4503bbb-283e-431a-beb8-250a18584206-metrics-certs" pod "f4503bbb-283e-431a-beb8-250a18584206" (UID: "f4503bbb-283e-431a-beb8-250a18584206"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:15:46.140882 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.140773    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4503bbb-283e-431a-beb8-250a18584206-webhook-cert" pod "f4503bbb-283e-431a-beb8-250a18584206" (UID: "f4503bbb-283e-431a-beb8-250a18584206"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:15:46.140882 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.140824    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4503bbb-283e-431a-beb8-250a18584206-kube-api-access-5hg6z" pod "f4503bbb-283e-431a-beb8-250a18584206" (UID: "f4503bbb-283e-431a-beb8-250a18584206"). InnerVolumeSpecName "kube-api-access-5hg6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:46.240381 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.240346    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hg6z\" (UniqueName: \"kubernetes.io/projected/f4503bbb-283e-431a-beb8-250a18584206-kube-api-access-5hg6z\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:46.240381 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.240376    4464 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4503bbb-283e-431a-beb8-250a18584206-metrics-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:46.240381 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.240386    4464 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f4503bbb-283e-431a-beb8-250a18584206-webhook-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:15:46.324531 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.324493    4464 generic.go:356] "Generic (PLEG): container finished" podID="f4503bbb-283e-431a-beb8-250a18584206" containerID="c0b24842bef74fbcdb7196d6a72cb94e703666927a4470bf78f90f8f406452ab" exitCode=0
Nov 02 13:15:46.324744 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.324538    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5975b94bc-2j9j5"
Nov 02 13:15:46.324744 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.324622    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5975b94bc-2j9j5" event={"ID":"f4503bbb-283e-431a-beb8-250a18584206","Type":"ContainerDied","Data":"c0b24842bef74fbcdb7196d6a72cb94e703666927a4470bf78f90f8f406452ab"}
Nov 02 13:15:46.324744 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.324722    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5975b94bc-2j9j5" event={"ID":"f4503bbb-283e-431a-beb8-250a18584206","Type":"ContainerDied","Data":"df46001fc93b4dea87e31c93d4c413d4e93bcc36fcb5144094d9c58d190a51c8"}
Nov 02 13:15:46.324943 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.324758    4464 scope.go:122] "RemoveContainer" containerID="c0b24842bef74fbcdb7196d6a72cb94e703666927a4470bf78f90f8f406452ab"
Nov 02 13:15:46.337575 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.337552    4464 scope.go:122] "RemoveContainer" containerID="c0b24842bef74fbcdb7196d6a72cb94e703666927a4470bf78f90f8f406452ab"
Nov 02 13:15:46.337905 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:15:46.337881    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b24842bef74fbcdb7196d6a72cb94e703666927a4470bf78f90f8f406452ab\": container with ID starting with c0b24842bef74fbcdb7196d6a72cb94e703666927a4470bf78f90f8f406452ab not found: ID does not exist" containerID="c0b24842bef74fbcdb7196d6a72cb94e703666927a4470bf78f90f8f406452ab"
Nov 02 13:15:46.337967 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.337908    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b24842bef74fbcdb7196d6a72cb94e703666927a4470bf78f90f8f406452ab"} err="failed to get container status \"c0b24842bef74fbcdb7196d6a72cb94e703666927a4470bf78f90f8f406452ab\": rpc error: code = NotFound desc = could not find container \"c0b24842bef74fbcdb7196d6a72cb94e703666927a4470bf78f90f8f406452ab\": container with ID starting with c0b24842bef74fbcdb7196d6a72cb94e703666927a4470bf78f90f8f406452ab not found: ID does not exist"
Nov 02 13:15:46.551744 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.551711    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ppqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Nov 02 13:15:46.551923 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.551759    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4" podUID="e5780a48-9777-4e56-87b4-e0d00bebb615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Nov 02 13:15:46.596001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.595054    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5975b94bc-2j9j5"]
Nov 02 13:15:46.609023 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.601616    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5975b94bc-2j9j5"]
Nov 02 13:15:46.905489 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.905396    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7xbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 02 13:15:46.906125 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:46.906090    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw" podUID="931c4f3d-1712-4c42-9226-ec630f19f7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 02 13:15:47.552428 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:47.552393    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ppqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Nov 02 13:15:47.552621 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:47.552449    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4" podUID="e5780a48-9777-4e56-87b4-e0d00bebb615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Nov 02 13:15:47.590998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:47.590148    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4503bbb-283e-431a-beb8-250a18584206" path="/var/lib/kubelet/pods/f4503bbb-283e-431a-beb8-250a18584206/volumes"
Nov 02 13:15:47.901628 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:47.901535    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7xbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 02 13:15:47.901794 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:47.901619    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw" podUID="931c4f3d-1712-4c42-9226-ec630f19f7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 02 13:15:47.976000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:47.968792    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5975b94bc-gwv4h"]
Nov 02 13:15:47.976000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:47.969467    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f4503bbb-283e-431a-beb8-250a18584206" containerName="manager"
Nov 02 13:15:47.976000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:47.969489    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4503bbb-283e-431a-beb8-250a18584206" containerName="manager"
Nov 02 13:15:47.976000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:47.969631    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4503bbb-283e-431a-beb8-250a18584206" containerName="manager"
Nov 02 13:15:47.990001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:47.983088    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5975b94bc-gwv4h"
Nov 02 13:15:48.037028 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.037002    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 13:15:48.056581 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.056542    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxhh4\" (UniqueName: \"kubernetes.io/projected/71cc9fa0-4ec8-499c-a857-9e7dffe12e63-kube-api-access-hxhh4\") pod \"maas-controller-5975b94bc-gwv4h\" (UID: \"71cc9fa0-4ec8-499c-a857-9e7dffe12e63\") " pod="opendatahub/maas-controller-5975b94bc-gwv4h"
Nov 02 13:15:48.056747 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.056590    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71cc9fa0-4ec8-499c-a857-9e7dffe12e63-webhook-cert\") pod \"maas-controller-5975b94bc-gwv4h\" (UID: \"71cc9fa0-4ec8-499c-a857-9e7dffe12e63\") " pod="opendatahub/maas-controller-5975b94bc-gwv4h"
Nov 02 13:15:48.056747 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.056694    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71cc9fa0-4ec8-499c-a857-9e7dffe12e63-metrics-certs\") pod \"maas-controller-5975b94bc-gwv4h\" (UID: \"71cc9fa0-4ec8-499c-a857-9e7dffe12e63\") " pod="opendatahub/maas-controller-5975b94bc-gwv4h"
Nov 02 13:15:48.092362 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.092261    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 13:15:48.092362 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.092307    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 13:15:48.092634 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.092378    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tzzlq\""
Nov 02 13:15:48.094526 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.094497    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5975b94bc-gwv4h"]
Nov 02 13:15:48.157915 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.157839    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71cc9fa0-4ec8-499c-a857-9e7dffe12e63-webhook-cert\") pod \"maas-controller-5975b94bc-gwv4h\" (UID: \"71cc9fa0-4ec8-499c-a857-9e7dffe12e63\") " pod="opendatahub/maas-controller-5975b94bc-gwv4h"
Nov 02 13:15:48.157915 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.157890    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71cc9fa0-4ec8-499c-a857-9e7dffe12e63-metrics-certs\") pod \"maas-controller-5975b94bc-gwv4h\" (UID: \"71cc9fa0-4ec8-499c-a857-9e7dffe12e63\") " pod="opendatahub/maas-controller-5975b94bc-gwv4h"
Nov 02 13:15:48.158137 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.157962    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxhh4\" (UniqueName: \"kubernetes.io/projected/71cc9fa0-4ec8-499c-a857-9e7dffe12e63-kube-api-access-hxhh4\") pod \"maas-controller-5975b94bc-gwv4h\" (UID: \"71cc9fa0-4ec8-499c-a857-9e7dffe12e63\") " pod="opendatahub/maas-controller-5975b94bc-gwv4h"
Nov 02 13:15:48.178625 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.178555    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71cc9fa0-4ec8-499c-a857-9e7dffe12e63-webhook-cert\") pod \"maas-controller-5975b94bc-gwv4h\" (UID: \"71cc9fa0-4ec8-499c-a857-9e7dffe12e63\") " pod="opendatahub/maas-controller-5975b94bc-gwv4h"
Nov 02 13:15:48.184588 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.184502    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71cc9fa0-4ec8-499c-a857-9e7dffe12e63-metrics-certs\") pod \"maas-controller-5975b94bc-gwv4h\" (UID: \"71cc9fa0-4ec8-499c-a857-9e7dffe12e63\") " pod="opendatahub/maas-controller-5975b94bc-gwv4h"
Nov 02 13:15:48.220635 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.220602    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxhh4\" (UniqueName: \"kubernetes.io/projected/71cc9fa0-4ec8-499c-a857-9e7dffe12e63-kube-api-access-hxhh4\") pod \"maas-controller-5975b94bc-gwv4h\" (UID: \"71cc9fa0-4ec8-499c-a857-9e7dffe12e63\") " pod="opendatahub/maas-controller-5975b94bc-gwv4h"
Nov 02 13:15:48.298811 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.298779    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5975b94bc-gwv4h"
Nov 02 13:15:48.552036 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.552007    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ppqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Nov 02 13:15:48.552227 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.552058    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4" podUID="e5780a48-9777-4e56-87b4-e0d00bebb615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Nov 02 13:15:48.683841 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.683812    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5975b94bc-gwv4h"]
Nov 02 13:15:48.685579 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:15:48.685553    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71cc9fa0_4ec8_499c_a857_9e7dffe12e63.slice/crio-401a65cd922ec44c012b7aafc91c8096abee110445695b7806e96dedcf312803 WatchSource:0}: Error finding container 401a65cd922ec44c012b7aafc91c8096abee110445695b7806e96dedcf312803: Status 404 returned error can't find the container with id 401a65cd922ec44c012b7aafc91c8096abee110445695b7806e96dedcf312803
Nov 02 13:15:48.901392 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.901295    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7xbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 02 13:15:48.901392 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:48.901370    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw" podUID="931c4f3d-1712-4c42-9226-ec630f19f7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 02 13:15:49.345216 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:49.345177    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5975b94bc-gwv4h" event={"ID":"71cc9fa0-4ec8-499c-a857-9e7dffe12e63","Type":"ContainerStarted","Data":"401a65cd922ec44c012b7aafc91c8096abee110445695b7806e96dedcf312803"}
Nov 02 13:15:49.552227 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:49.552204    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ppqp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body=
Nov 02 13:15:49.552342 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:49.552253    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4" podUID="e5780a48-9777-4e56-87b4-e0d00bebb615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused"
Nov 02 13:15:49.901216 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:49.901179    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7xbnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Nov 02 13:15:49.901419 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:49.901246    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw" podUID="931c4f3d-1712-4c42-9226-ec630f19f7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Nov 02 13:15:50.353672 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:50.353625    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5975b94bc-gwv4h" event={"ID":"71cc9fa0-4ec8-499c-a857-9e7dffe12e63","Type":"ContainerStarted","Data":"976c86afbc5996423cf7737d41abfc960e764b4afdd395eac11431aaf1932359"}
Nov 02 13:15:50.354227 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:50.353814    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5975b94bc-gwv4h"
Nov 02 13:15:50.523421 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:50.523352    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5975b94bc-gwv4h" podStartSLOduration=2.710358592 podStartE2EDuration="3.523332238s" totalImagesPullingTime="812.973646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:50.517739069 +0000 UTC m=+4779.081447485" watchObservedRunningTime="2026-11-02 13:15:50.523332238 +0000 UTC m=+4779.087040650"
Nov 02 13:15:51.322009 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:51.321940    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:51.322777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:51.322752    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:51.323689 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:51.323669    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:15:51.623255 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:51.622610    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:51.623796 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:51.623777    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:51.624650 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:15:51.624627    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:15:58.291431 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:15:58.291394    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:16:01.362545 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:01.362518    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5975b94bc-gwv4h"
Nov 02 13:16:12.202168 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:12.202142    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:16:12.218709 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:12.218687    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:16:12.244732 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:12.244709    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:16:12.260767 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:12.260744    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:16:15.199577 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:15.199537    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:16:26.612634 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.612547    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"]
Nov 02 13:16:26.617909 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.617884    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.724285 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.724252    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.724485 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.724306    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.724485 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.724409    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.724485 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.724449    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.724485 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.724473    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.724733 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.724620    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.724733 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.724664    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjp8k\" (UniqueName: \"kubernetes.io/projected/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-kube-api-access-vjp8k\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.724853 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.724788    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.724853 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.724817    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.825711 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.825674    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.825711 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.825712    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.826000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.825742    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.826000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.825855    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.826000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.825925    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.826299 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.826259    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.827012 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.826989    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.827744 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.827717    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.828219 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.828193    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.828435 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.828413    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.828592 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.828576    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.828730 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.828713    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjp8k\" (UniqueName: \"kubernetes.io/projected/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-kube-api-access-vjp8k\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.829003 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.828949    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.829243 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.829219    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.830793 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.830742    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"]
Nov 02 13:16:26.831658 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.831633    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.832821 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.832798    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.888062 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.887964    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.890654 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.890624    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjp8k\" (UniqueName: \"kubernetes.io/projected/bf09d973-3e5b-4daa-aa7f-cf839b61d1df-kube-api-access-vjp8k\") pod \"maas-default-gateway-openshift-default-6dcf757899-gt29z\" (UID: \"bf09d973-3e5b-4daa-aa7f-cf839b61d1df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:26.930218 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:26.930185    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:27.356569 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:27.356537    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"]
Nov 02 13:16:27.390779 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:16:27.390753    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf09d973_3e5b_4daa_aa7f_cf839b61d1df.slice/crio-a88572178f4b42138028c10a039c61cab2d91aea1b48618c5417b239e15b8c42 WatchSource:0}: Error finding container a88572178f4b42138028c10a039c61cab2d91aea1b48618c5417b239e15b8c42: Status 404 returned error can't find the container with id a88572178f4b42138028c10a039c61cab2d91aea1b48618c5417b239e15b8c42
Nov 02 13:16:27.392961 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:27.392931    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:16:27.393066 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:27.393014    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:16:27.393066 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:27.393044    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:16:27.549405 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:27.549359    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" event={"ID":"bf09d973-3e5b-4daa-aa7f-cf839b61d1df","Type":"ContainerStarted","Data":"2b6b4eec7ca2de4b1df534ab86b9188446329e5f67e03bdb70c7c74be0beeabc"}
Nov 02 13:16:27.549603 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:27.549414    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" event={"ID":"bf09d973-3e5b-4daa-aa7f-cf839b61d1df","Type":"ContainerStarted","Data":"a88572178f4b42138028c10a039c61cab2d91aea1b48618c5417b239e15b8c42"}
Nov 02 13:16:27.672536 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:27.672431    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podStartSLOduration=1.67241732 podStartE2EDuration="1.67241732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:16:27.671796194 +0000 UTC m=+4816.235504603" watchObservedRunningTime="2026-11-02 13:16:27.67241732 +0000 UTC m=+4816.236125728"
Nov 02 13:16:27.930806 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:27.930720    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:28.308763 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:16:28.308729    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:16:28.932050 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:28.932013    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gt29z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Nov 02 13:16:28.932477 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:28.932067    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podUID="bf09d973-3e5b-4daa-aa7f-cf839b61d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Nov 02 13:16:29.931214 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:29.931175    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gt29z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Nov 02 13:16:29.931406 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:29.931236    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podUID="bf09d973-3e5b-4daa-aa7f-cf839b61d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Nov 02 13:16:30.930726 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:30.930691    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gt29z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Nov 02 13:16:30.931172 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:30.930746    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podUID="bf09d973-3e5b-4daa-aa7f-cf839b61d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Nov 02 13:16:31.930832 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:31.930798    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gt29z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Nov 02 13:16:31.931261 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:31.930855    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podUID="bf09d973-3e5b-4daa-aa7f-cf839b61d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Nov 02 13:16:32.931232 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:32.931192    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gt29z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Nov 02 13:16:32.931651 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:32.931251    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podUID="bf09d973-3e5b-4daa-aa7f-cf839b61d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Nov 02 13:16:33.931172 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:33.931136    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gt29z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Nov 02 13:16:33.931363 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:33.931187    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podUID="bf09d973-3e5b-4daa-aa7f-cf839b61d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Nov 02 13:16:34.931230 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:34.931197    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gt29z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Nov 02 13:16:34.931403 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:34.931245    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podUID="bf09d973-3e5b-4daa-aa7f-cf839b61d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Nov 02 13:16:35.001252 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:35.001223    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:16:36.931748 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:36.931707    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gt29z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:16:36.932284 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:36.931765    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podUID="bf09d973-3e5b-4daa-aa7f-cf839b61d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:16:37.932533 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:37.932487    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gt29z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:16:37.932967 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:37.932541    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podUID="bf09d973-3e5b-4daa-aa7f-cf839b61d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:16:38.974184 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:38.974133    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gt29z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:16:38.974744 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:38.974224    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podUID="bf09d973-3e5b-4daa-aa7f-cf839b61d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:16:40.017107 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:40.017070    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gt29z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:16:40.017526 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:40.017130    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podUID="bf09d973-3e5b-4daa-aa7f-cf839b61d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:16:40.077932 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:40.077904    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:40.078242 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:40.078212    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:40.078848 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:40.078826    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z"
Nov 02 13:16:45.047947 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:45.047908    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"]
Nov 02 13:16:45.098434 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:45.098404    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd03fbd1-c6kcd"]
Nov 02 13:16:45.301265 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:45.301186    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:16:45.581044 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:45.580931    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407a7564-7d06-4f2a-a433-9e28c23e8550" path="/var/lib/kubelet/pods/407a7564-7d06-4f2a-a433-9e28c23e8550/volumes"
Nov 02 13:16:55.664896 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:55.664861    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:16:57.048566 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.048524    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"]
Nov 02 13:16:57.098826 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.098794    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41b2555-cztmt"]
Nov 02 13:16:57.293635 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.293599    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"]
Nov 02 13:16:57.297166 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.297141    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"
Nov 02 13:16:57.352148 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.352035    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:16:57.352331 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.352180    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:16:57.352331 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.352192    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:16:57.352331 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.352234    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:16:57.355444 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.355416    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"]
Nov 02 13:16:57.383725 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.383698    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-h4z29\" (UID: \"2634b75b-1e7c-46f1-9f54-437fdd72dd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"
Nov 02 13:16:57.383864 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.383816    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-h4z29\" (UID: \"2634b75b-1e7c-46f1-9f54-437fdd72dd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"
Nov 02 13:16:57.383919 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.383877    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5klp\" (UniqueName: \"kubernetes.io/projected/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-kube-api-access-c5klp\") pod \"maas-api-revoke-sub-f03629db-h4z29\" (UID: \"2634b75b-1e7c-46f1-9f54-437fdd72dd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"
Nov 02 13:16:57.485327 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.485296    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-h4z29\" (UID: \"2634b75b-1e7c-46f1-9f54-437fdd72dd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"
Nov 02 13:16:57.485648 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.485366    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5klp\" (UniqueName: \"kubernetes.io/projected/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-kube-api-access-c5klp\") pod \"maas-api-revoke-sub-f03629db-h4z29\" (UID: \"2634b75b-1e7c-46f1-9f54-437fdd72dd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"
Nov 02 13:16:57.485648 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.485412    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-h4z29\" (UID: \"2634b75b-1e7c-46f1-9f54-437fdd72dd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"
Nov 02 13:16:57.486140 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.486116    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-h4z29\" (UID: \"2634b75b-1e7c-46f1-9f54-437fdd72dd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"
Nov 02 13:16:57.487806 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.487776    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-h4z29\" (UID: \"2634b75b-1e7c-46f1-9f54-437fdd72dd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"
Nov 02 13:16:57.547453 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.547422    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5klp\" (UniqueName: \"kubernetes.io/projected/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-kube-api-access-c5klp\") pod \"maas-api-revoke-sub-f03629db-h4z29\" (UID: \"2634b75b-1e7c-46f1-9f54-437fdd72dd8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"
Nov 02 13:16:57.580925 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.580843    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e1a02f-bf60-4321-9bd4-1313a8c4fd55" path="/var/lib/kubelet/pods/64e1a02f-bf60-4321-9bd4-1313a8c4fd55/volumes"
Nov 02 13:16:57.610191 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.610136    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"
Nov 02 13:16:57.983255 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:57.983216    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"]
Nov 02 13:16:57.991336 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:16:57.991308    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2634b75b_1e7c_46f1_9f54_437fdd72dd8c.slice/crio-97a241884cb3fbda007ffca7dbf256a905260580f08a1071697cb64212aba2f5 WatchSource:0}: Error finding container 97a241884cb3fbda007ffca7dbf256a905260580f08a1071697cb64212aba2f5: Status 404 returned error can't find the container with id 97a241884cb3fbda007ffca7dbf256a905260580f08a1071697cb64212aba2f5
Nov 02 13:16:58.290509 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:16:58.290479    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:16:58.768096 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:58.768063    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29" event={"ID":"2634b75b-1e7c-46f1-9f54-437fdd72dd8c","Type":"ContainerStarted","Data":"3d871723a4fc6992cb6cc33cd35892ce13cfe273e94ca4e84d1b66d23bfde032"}
Nov 02 13:16:58.768289 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:58.768119    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29" event={"ID":"2634b75b-1e7c-46f1-9f54-437fdd72dd8c","Type":"ContainerStarted","Data":"97a241884cb3fbda007ffca7dbf256a905260580f08a1071697cb64212aba2f5"}
Nov 02 13:16:58.886233 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:58.886191    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29" podStartSLOduration=1.886174605 podStartE2EDuration="1.886174605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:16:58.88550187 +0000 UTC m=+4847.449210274" watchObservedRunningTime="2026-11-02 13:16:58.886174605 +0000 UTC m=+4847.449883040"
Nov 02 13:16:59.774910 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:59.774874    4464 generic.go:356] "Generic (PLEG): container finished" podID="2634b75b-1e7c-46f1-9f54-437fdd72dd8c" containerID="3d871723a4fc6992cb6cc33cd35892ce13cfe273e94ca4e84d1b66d23bfde032" exitCode=0
Nov 02 13:16:59.775318 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:16:59.774928    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29" event={"ID":"2634b75b-1e7c-46f1-9f54-437fdd72dd8c","Type":"ContainerDied","Data":"3d871723a4fc6992cb6cc33cd35892ce13cfe273e94ca4e84d1b66d23bfde032"}
Nov 02 13:17:00.912701 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:00.912680    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"
Nov 02 13:17:01.013629 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:01.013602    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-maas-api-xks-ca\") pod \"2634b75b-1e7c-46f1-9f54-437fdd72dd8c\" (UID: \"2634b75b-1e7c-46f1-9f54-437fdd72dd8c\") "
Nov 02 13:17:01.013764 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:01.013655    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-maas-api-openshift-ca\") pod \"2634b75b-1e7c-46f1-9f54-437fdd72dd8c\" (UID: \"2634b75b-1e7c-46f1-9f54-437fdd72dd8c\") "
Nov 02 13:17:01.013764 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:01.013688    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-kube-api-access-c5klp\" (UniqueName: \"kubernetes.io/projected/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-kube-api-access-c5klp\") pod \"2634b75b-1e7c-46f1-9f54-437fdd72dd8c\" (UID: \"2634b75b-1e7c-46f1-9f54-437fdd72dd8c\") "
Nov 02 13:17:01.014024 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:01.014002    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-maas-api-openshift-ca" pod "2634b75b-1e7c-46f1-9f54-437fdd72dd8c" (UID: "2634b75b-1e7c-46f1-9f54-437fdd72dd8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:17:01.015569 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:01.015548    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-maas-api-xks-ca" pod "2634b75b-1e7c-46f1-9f54-437fdd72dd8c" (UID: "2634b75b-1e7c-46f1-9f54-437fdd72dd8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:17:01.015769 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:01.015751    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-kube-api-access-c5klp" pod "2634b75b-1e7c-46f1-9f54-437fdd72dd8c" (UID: "2634b75b-1e7c-46f1-9f54-437fdd72dd8c"). InnerVolumeSpecName "kube-api-access-c5klp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:17:01.114710 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:01.114640    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:17:01.114710 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:01.114666    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:17:01.114710 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:01.114678    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5klp\" (UniqueName: \"kubernetes.io/projected/2634b75b-1e7c-46f1-9f54-437fdd72dd8c-kube-api-access-c5klp\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:17:01.787018 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:01.786991    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"
Nov 02 13:17:01.787018 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:01.786999    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29" event={"ID":"2634b75b-1e7c-46f1-9f54-437fdd72dd8c","Type":"ContainerDied","Data":"97a241884cb3fbda007ffca7dbf256a905260580f08a1071697cb64212aba2f5"}
Nov 02 13:17:01.787231 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:01.787041    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97a241884cb3fbda007ffca7dbf256a905260580f08a1071697cb64212aba2f5"
Nov 02 13:17:04.515620 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:04.515588    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:17:14.048492 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:14.048456    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"]
Nov 02 13:17:14.099704 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:14.099662    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5af16306-dmbvs"]
Nov 02 13:17:15.612302 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:15.612257    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17410b5d-cf59-4ef2-90f9-e10d8bea329f" path="/var/lib/kubelet/pods/17410b5d-cf59-4ef2-90f9-e10d8bea329f/volumes"
Nov 02 13:17:19.032735 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.032701    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"]
Nov 02 13:17:19.033171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.033158    4464 cpu_manager.go:414] "containerMap: removing container" podUID="2634b75b-1e7c-46f1-9f54-437fdd72dd8c" containerName="revoke-keys"
Nov 02 13:17:19.033222 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.033172    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2634b75b-1e7c-46f1-9f54-437fdd72dd8c" containerName="revoke-keys"
Nov 02 13:17:19.033259 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.033239    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="2634b75b-1e7c-46f1-9f54-437fdd72dd8c" containerName="revoke-keys"
Nov 02 13:17:19.037526 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.037510    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"
Nov 02 13:17:19.095331 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.095300    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"]
Nov 02 13:17:19.124180 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.124154    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qslbh\" (UniqueName: \"kubernetes.io/projected/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-kube-api-access-qslbh\") pod \"maas-api-revoke-sub-0165d7d2-7sq46\" (UID: \"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"
Nov 02 13:17:19.124333 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.124186    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7sq46\" (UID: \"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"
Nov 02 13:17:19.124333 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.124273    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7sq46\" (UID: \"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"
Nov 02 13:17:19.142448 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.142424    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:17:19.142582 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.142424    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:17:19.142582 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.142424    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:17:19.142582 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.142427    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:17:19.224905 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.224881    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7sq46\" (UID: \"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"
Nov 02 13:17:19.225067 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.224996    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qslbh\" (UniqueName: \"kubernetes.io/projected/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-kube-api-access-qslbh\") pod \"maas-api-revoke-sub-0165d7d2-7sq46\" (UID: \"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"
Nov 02 13:17:19.225067 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.225014    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7sq46\" (UID: \"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"
Nov 02 13:17:19.225552 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.225534    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7sq46\" (UID: \"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"
Nov 02 13:17:19.227159 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.227141    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7sq46\" (UID: \"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"
Nov 02 13:17:19.287319 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.287257    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qslbh\" (UniqueName: \"kubernetes.io/projected/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-kube-api-access-qslbh\") pod \"maas-api-revoke-sub-0165d7d2-7sq46\" (UID: \"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"
Nov 02 13:17:19.350096 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.350068    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"
Nov 02 13:17:19.723121 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.723093    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"]
Nov 02 13:17:19.724883 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:17:19.724857    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac77eb8_8a43_430a_9ffd_6ad3a96d6c45.slice/crio-281389c67d631042bef67d1019913b765641aaad47c46ed30eab144001fb1c63 WatchSource:0}: Error finding container 281389c67d631042bef67d1019913b765641aaad47c46ed30eab144001fb1c63: Status 404 returned error can't find the container with id 281389c67d631042bef67d1019913b765641aaad47c46ed30eab144001fb1c63
Nov 02 13:17:19.900057 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.900027    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46" event={"ID":"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45","Type":"ContainerStarted","Data":"2c7a279d7240198af2e3df14058b9898aeaaf3fa93cb7195b63cb814133b7dfa"}
Nov 02 13:17:19.900170 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:19.900075    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46" event={"ID":"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45","Type":"ContainerStarted","Data":"281389c67d631042bef67d1019913b765641aaad47c46ed30eab144001fb1c63"}
Nov 02 13:17:20.017605 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:20.017500    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46" podStartSLOduration=2.017485263 podStartE2EDuration="2.017485263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:17:20.017213287 +0000 UTC m=+4868.580921697" watchObservedRunningTime="2026-11-02 13:17:20.017485263 +0000 UTC m=+4868.581193673"
Nov 02 13:17:20.906644 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:20.906605    4464 generic.go:356] "Generic (PLEG): container finished" podID="0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45" containerID="2c7a279d7240198af2e3df14058b9898aeaaf3fa93cb7195b63cb814133b7dfa" exitCode=0
Nov 02 13:17:20.907057 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:20.906683    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46" event={"ID":"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45","Type":"ContainerDied","Data":"2c7a279d7240198af2e3df14058b9898aeaaf3fa93cb7195b63cb814133b7dfa"}
Nov 02 13:17:22.045283 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:22.045259    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"
Nov 02 13:17:22.157068 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:22.157041    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-maas-api-xks-ca\") pod \"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45\" (UID: \"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45\") "
Nov 02 13:17:22.157185 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:22.157085    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-kube-api-access-qslbh\" (UniqueName: \"kubernetes.io/projected/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-kube-api-access-qslbh\") pod \"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45\" (UID: \"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45\") "
Nov 02 13:17:22.157232 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:22.157208    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-maas-api-openshift-ca\") pod \"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45\" (UID: \"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45\") "
Nov 02 13:17:22.157535 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:22.157507    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-maas-api-openshift-ca" pod "0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45" (UID: "0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:17:22.159011 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:22.158989    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-maas-api-xks-ca" pod "0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45" (UID: "0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:17:22.159105 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:22.159082    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-kube-api-access-qslbh" pod "0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45" (UID: "0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45"). InnerVolumeSpecName "kube-api-access-qslbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:17:22.258734 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:22.258709    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:17:22.258734 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:22.258732    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qslbh\" (UniqueName: \"kubernetes.io/projected/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-kube-api-access-qslbh\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:17:22.258891 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:22.258743    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:17:22.917633 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:22.917599    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"
Nov 02 13:17:22.917794 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:22.917628    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46" event={"ID":"0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45","Type":"ContainerDied","Data":"281389c67d631042bef67d1019913b765641aaad47c46ed30eab144001fb1c63"}
Nov 02 13:17:22.917794 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:22.917686    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="281389c67d631042bef67d1019913b765641aaad47c46ed30eab144001fb1c63"
Nov 02 13:17:25.064289 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:25.064256    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"]
Nov 02 13:17:25.115052 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:25.115018    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59e625b-ggdjv"]
Nov 02 13:17:25.581231 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:25.581199    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae14ab2-1a37-440e-ad0b-0fb57a7ef755" path="/var/lib/kubelet/pods/bae14ab2-1a37-440e-ad0b-0fb57a7ef755/volumes"
Nov 02 13:17:28.290927 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:17:28.290890    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:17:30.785255 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.785221    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"]
Nov 02 13:17:30.785670 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.785655    4464 cpu_manager.go:414] "containerMap: removing container" podUID="0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45" containerName="revoke-keys"
Nov 02 13:17:30.785670 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.785666    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45" containerName="revoke-keys"
Nov 02 13:17:30.785758 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.785748    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45" containerName="revoke-keys"
Nov 02 13:17:30.789007 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.788992    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"
Nov 02 13:17:30.835655 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.835622    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b19b839-2de9-48f4-9877-6d823d442d2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9m49s\" (UID: \"8b19b839-2de9-48f4-9877-6d823d442d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"
Nov 02 13:17:30.835787 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.835673    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cknq2\" (UniqueName: \"kubernetes.io/projected/8b19b839-2de9-48f4-9877-6d823d442d2b-kube-api-access-cknq2\") pod \"maas-api-revoke-sub-6b19338f-9m49s\" (UID: \"8b19b839-2de9-48f4-9877-6d823d442d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"
Nov 02 13:17:30.835787 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.835736    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b19b839-2de9-48f4-9877-6d823d442d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9m49s\" (UID: \"8b19b839-2de9-48f4-9877-6d823d442d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"
Nov 02 13:17:30.844508 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.844487    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:17:30.844625 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.844611    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:17:30.895117 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.895090    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:17:30.895262 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.895145    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:17:30.897367 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.897347    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"]
Nov 02 13:17:30.936562 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.936530    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cknq2\" (UniqueName: \"kubernetes.io/projected/8b19b839-2de9-48f4-9877-6d823d442d2b-kube-api-access-cknq2\") pod \"maas-api-revoke-sub-6b19338f-9m49s\" (UID: \"8b19b839-2de9-48f4-9877-6d823d442d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"
Nov 02 13:17:30.936712 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.936580    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b19b839-2de9-48f4-9877-6d823d442d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9m49s\" (UID: \"8b19b839-2de9-48f4-9877-6d823d442d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"
Nov 02 13:17:30.936712 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.936694    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b19b839-2de9-48f4-9877-6d823d442d2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9m49s\" (UID: \"8b19b839-2de9-48f4-9877-6d823d442d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"
Nov 02 13:17:30.937255 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.937232    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b19b839-2de9-48f4-9877-6d823d442d2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9m49s\" (UID: \"8b19b839-2de9-48f4-9877-6d823d442d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"
Nov 02 13:17:30.939007 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.938986    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b19b839-2de9-48f4-9877-6d823d442d2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9m49s\" (UID: \"8b19b839-2de9-48f4-9877-6d823d442d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"
Nov 02 13:17:30.999056 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:30.999032    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cknq2\" (UniqueName: \"kubernetes.io/projected/8b19b839-2de9-48f4-9877-6d823d442d2b-kube-api-access-cknq2\") pod \"maas-api-revoke-sub-6b19338f-9m49s\" (UID: \"8b19b839-2de9-48f4-9877-6d823d442d2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"
Nov 02 13:17:31.102149 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:31.102084    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"
Nov 02 13:17:31.473539 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:31.473499    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"]
Nov 02 13:17:31.474721 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:17:31.474692    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b19b839_2de9_48f4_9877_6d823d442d2b.slice/crio-444bfd02acb62ea9c75c144c7b7b905458773366c52edc3c7358cdb99c9a2345 WatchSource:0}: Error finding container 444bfd02acb62ea9c75c144c7b7b905458773366c52edc3c7358cdb99c9a2345: Status 404 returned error can't find the container with id 444bfd02acb62ea9c75c144c7b7b905458773366c52edc3c7358cdb99c9a2345
Nov 02 13:17:31.962318 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:31.962281    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s" event={"ID":"8b19b839-2de9-48f4-9877-6d823d442d2b","Type":"ContainerStarted","Data":"e08307775246c1ad69f11e51e17a194fad04ef330eb681d8ba59525692b237ee"}
Nov 02 13:17:31.962681 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:31.962337    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s" event={"ID":"8b19b839-2de9-48f4-9877-6d823d442d2b","Type":"ContainerStarted","Data":"444bfd02acb62ea9c75c144c7b7b905458773366c52edc3c7358cdb99c9a2345"}
Nov 02 13:17:32.082775 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:32.082685    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s" podStartSLOduration=2.082672441 podStartE2EDuration="2.082672441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:17:32.081055188 +0000 UTC m=+4880.644763596" watchObservedRunningTime="2026-11-02 13:17:32.082672441 +0000 UTC m=+4880.646380849"
Nov 02 13:17:32.971787 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:32.971750    4464 generic.go:356] "Generic (PLEG): container finished" podID="8b19b839-2de9-48f4-9877-6d823d442d2b" containerID="e08307775246c1ad69f11e51e17a194fad04ef330eb681d8ba59525692b237ee" exitCode=0
Nov 02 13:17:32.971787 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:32.971790    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s" event={"ID":"8b19b839-2de9-48f4-9877-6d823d442d2b","Type":"ContainerDied","Data":"e08307775246c1ad69f11e51e17a194fad04ef330eb681d8ba59525692b237ee"}
Nov 02 13:17:34.113262 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:34.113233    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"
Nov 02 13:17:34.169016 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:34.168971    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b19b839-2de9-48f4-9877-6d823d442d2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b19b839-2de9-48f4-9877-6d823d442d2b-maas-api-xks-ca\") pod \"8b19b839-2de9-48f4-9877-6d823d442d2b\" (UID: \"8b19b839-2de9-48f4-9877-6d823d442d2b\") "
Nov 02 13:17:34.169138 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:34.169040    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b19b839-2de9-48f4-9877-6d823d442d2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b19b839-2de9-48f4-9877-6d823d442d2b-maas-api-openshift-ca\") pod \"8b19b839-2de9-48f4-9877-6d823d442d2b\" (UID: \"8b19b839-2de9-48f4-9877-6d823d442d2b\") "
Nov 02 13:17:34.169203 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:34.169185    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b19b839-2de9-48f4-9877-6d823d442d2b-kube-api-access-cknq2\" (UniqueName: \"kubernetes.io/projected/8b19b839-2de9-48f4-9877-6d823d442d2b-kube-api-access-cknq2\") pod \"8b19b839-2de9-48f4-9877-6d823d442d2b\" (UID: \"8b19b839-2de9-48f4-9877-6d823d442d2b\") "
Nov 02 13:17:34.169447 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:34.169412    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b19b839-2de9-48f4-9877-6d823d442d2b-maas-api-openshift-ca" pod "8b19b839-2de9-48f4-9877-6d823d442d2b" (UID: "8b19b839-2de9-48f4-9877-6d823d442d2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:17:34.171004 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:34.170965    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b19b839-2de9-48f4-9877-6d823d442d2b-maas-api-xks-ca" pod "8b19b839-2de9-48f4-9877-6d823d442d2b" (UID: "8b19b839-2de9-48f4-9877-6d823d442d2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:17:34.171224 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:34.171194    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b19b839-2de9-48f4-9877-6d823d442d2b-kube-api-access-cknq2" pod "8b19b839-2de9-48f4-9877-6d823d442d2b" (UID: "8b19b839-2de9-48f4-9877-6d823d442d2b"). InnerVolumeSpecName "kube-api-access-cknq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:17:34.270877 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:34.270810    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cknq2\" (UniqueName: \"kubernetes.io/projected/8b19b839-2de9-48f4-9877-6d823d442d2b-kube-api-access-cknq2\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:17:34.270877 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:34.270838    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b19b839-2de9-48f4-9877-6d823d442d2b-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:17:34.270877 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:34.270849    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b19b839-2de9-48f4-9877-6d823d442d2b-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:17:34.982967 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:34.982932    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s" event={"ID":"8b19b839-2de9-48f4-9877-6d823d442d2b","Type":"ContainerDied","Data":"444bfd02acb62ea9c75c144c7b7b905458773366c52edc3c7358cdb99c9a2345"}
Nov 02 13:17:34.982967 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:34.982957    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"
Nov 02 13:17:34.983201 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:34.983000    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="444bfd02acb62ea9c75c144c7b7b905458773366c52edc3c7358cdb99c9a2345"
Nov 02 13:17:39.048837 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:39.048808    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"]
Nov 02 13:17:39.100162 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:39.100128    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-338a0860-vgdd7"]
Nov 02 13:17:39.581025 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:39.580962    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bca2d4b2-04bd-4c4f-be95-223f0718393c" path="/var/lib/kubelet/pods/bca2d4b2-04bd-4c4f-be95-223f0718393c/volumes"
Nov 02 13:17:43.588420 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:43.588389    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:17:45.089783 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.089744    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"]
Nov 02 13:17:45.091132 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.091108    4464 cpu_manager.go:414] "containerMap: removing container" podUID="8b19b839-2de9-48f4-9877-6d823d442d2b" containerName="revoke-keys"
Nov 02 13:17:45.091293 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.091277    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b19b839-2de9-48f4-9877-6d823d442d2b" containerName="revoke-keys"
Nov 02 13:17:45.091502 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.091485    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b19b839-2de9-48f4-9877-6d823d442d2b" containerName="revoke-keys"
Nov 02 13:17:45.100003 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.096301    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.200004 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.199725    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41485cb-1f46-4207-9e2f-20e1f05f2502-tls-certs\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.200004 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.199793    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-dshm\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.200004 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.199840    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzdq\" (UniqueName: \"kubernetes.io/projected/e41485cb-1f46-4207-9e2f-20e1f05f2502-kube-api-access-mnzdq\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.200004 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.199902    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-home\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.200004 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.200007    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-tmp-dir\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.200391 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.200036    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-model-cache\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.242004 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.241610    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"]
Nov 02 13:17:45.242004 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.241852    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-b2fa3bcb-kserve-self-signed-certs\""
Nov 02 13:17:45.329457 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.329418    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-model-cache\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.330477 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.330449    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-model-cache\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.330790 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.330771    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41485cb-1f46-4207-9e2f-20e1f05f2502-tls-certs\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.331246 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.331129    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-dshm\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.331920 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.331498    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzdq\" (UniqueName: \"kubernetes.io/projected/e41485cb-1f46-4207-9e2f-20e1f05f2502-kube-api-access-mnzdq\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.332439 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.332422    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-home\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.333042 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.333025    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-tmp-dir\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.333642 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.333410    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-tmp-dir\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.333780 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.333554    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-home\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.342048 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.340266    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41485cb-1f46-4207-9e2f-20e1f05f2502-tls-certs\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.342048 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.340539    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-dshm\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.416051 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.415992    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzdq\" (UniqueName: \"kubernetes.io/projected/e41485cb-1f46-4207-9e2f-20e1f05f2502-kube-api-access-mnzdq\") pod \"e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.441878 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.441844    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:45.825001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.823881    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"]
Nov 02 13:17:45.838020 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.835890    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"
Nov 02 13:17:45.847666 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.847643    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m94sh\" (UniqueName: \"kubernetes.io/projected/bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9-kube-api-access-m94sh\") pod \"e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4\" (UID: \"bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"
Nov 02 13:17:45.847802 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.847698    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9-tls-certs\") pod \"e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4\" (UID: \"bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"
Nov 02 13:17:45.886996 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.886891    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"]
Nov 02 13:17:45.909537 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.909185    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-b2fa3bcb-epp-sa-dockercfg-rh99w\""
Nov 02 13:17:45.948001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.943824    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"]
Nov 02 13:17:45.952304 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.951602    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m94sh\" (UniqueName: \"kubernetes.io/projected/bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9-kube-api-access-m94sh\") pod \"e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4\" (UID: \"bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"
Nov 02 13:17:45.952304 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.951695    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9-tls-certs\") pod \"e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4\" (UID: \"bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"
Nov 02 13:17:45.954451 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:17:45.954426    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode41485cb_1f46_4207_9e2f_20e1f05f2502.slice/crio-ed31ed8fa06372a7aeb3f3a8aaabdd3127bfe83d613973c1bcfb07b1e5c8bd8e WatchSource:0}: Error finding container ed31ed8fa06372a7aeb3f3a8aaabdd3127bfe83d613973c1bcfb07b1e5c8bd8e: Status 404 returned error can't find the container with id ed31ed8fa06372a7aeb3f3a8aaabdd3127bfe83d613973c1bcfb07b1e5c8bd8e
Nov 02 13:17:45.955309 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:45.955287    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9-tls-certs\") pod \"e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4\" (UID: \"bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"
Nov 02 13:17:46.042011 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:46.032674    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m94sh\" (UniqueName: \"kubernetes.io/projected/bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9-kube-api-access-m94sh\") pod \"e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4\" (UID: \"bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9\") " pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"
Nov 02 13:17:46.055189 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:46.055149    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn" event={"ID":"e41485cb-1f46-4207-9e2f-20e1f05f2502","Type":"ContainerStarted","Data":"ed31ed8fa06372a7aeb3f3a8aaabdd3127bfe83d613973c1bcfb07b1e5c8bd8e"}
Nov 02 13:17:46.150590 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:46.150525    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"
Nov 02 13:17:46.706548 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:46.706519    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"]
Nov 02 13:17:46.708847 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:17:46.708820    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdedd10e_21ed_44f6_a1d9_e755b1f0d8d9.slice/crio-f451cda5e39185451cbc9e03c7a73dddc725d7c4bec232977e2a49a4c4d162b4 WatchSource:0}: Error finding container f451cda5e39185451cbc9e03c7a73dddc725d7c4bec232977e2a49a4c4d162b4: Status 404 returned error can't find the container with id f451cda5e39185451cbc9e03c7a73dddc725d7c4bec232977e2a49a4c4d162b4
Nov 02 13:17:47.061439 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:47.061399    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn" event={"ID":"e41485cb-1f46-4207-9e2f-20e1f05f2502","Type":"ContainerStarted","Data":"c23585430013a6d482b30891a44122a5aa6b8692ddb8bc90a334a32409f33011"}
Nov 02 13:17:47.062586 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:47.062566    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4" event={"ID":"bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9","Type":"ContainerStarted","Data":"f451cda5e39185451cbc9e03c7a73dddc725d7c4bec232977e2a49a4c4d162b4"}
Nov 02 13:17:47.185055 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:47.185008    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn" podStartSLOduration=2.184988583 podStartE2EDuration="2.184988583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:17:47.183301179 +0000 UTC m=+4895.747009588" watchObservedRunningTime="2026-11-02 13:17:47.184988583 +0000 UTC m=+4895.748696986"
Nov 02 13:17:47.443120 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:47.443037    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:49.075563 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:49.075525    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4" event={"ID":"bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9","Type":"ContainerStarted","Data":"489a96837f404967e6539fb90a24033b836b61d371cc76884aba4aaa3684f462"}
Nov 02 13:17:49.075942 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:49.075842    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"
Nov 02 13:17:49.195558 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:49.195469    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4" podStartSLOduration=2.031855318 podStartE2EDuration="4.195456733s" totalImagesPullingTime="2.163601415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:17:49.194430828 +0000 UTC m=+4897.758139235" watchObservedRunningTime="2026-11-02 13:17:49.195456733 +0000 UTC m=+4897.759165150"
Nov 02 13:17:51.052499 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:51.052465    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"]
Nov 02 13:17:51.101698 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:51.101652    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ca2812-9pnlj"]
Nov 02 13:17:51.584342 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:51.584304    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4620cb6f-7c27-4e6a-b2a2-caea0b4976f0" path="/var/lib/kubelet/pods/4620cb6f-7c27-4e6a-b2a2-caea0b4976f0/volumes"
Nov 02 13:17:55.442748 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:55.442669    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:55.454946 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:55.454920    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:56.164778 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:56.164747    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:17:58.201129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:17:58.201094    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:17:58.291536 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:17:58.291498    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:18:14.624442 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:18:14.624411    4464 scope.go:122] "RemoveContainer" containerID="326d1632c6d93644ef7ed4624897af7a710f9898f25c6a502a007bbffe8d4992"
Nov 02 13:18:14.633215 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:18:14.633195    4464 scope.go:122] "RemoveContainer" containerID="14b159824bc37024ea1d387506b9a5e98f169b4d30e362e4c44de98b57640746"
Nov 02 13:18:14.641358 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:18:14.641338    4464 scope.go:122] "RemoveContainer" containerID="7b4919aa105f576ac6ae0169c510c50c09a9ff968645d062ad57c402ac267948"
Nov 02 13:18:20.082206 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:18:20.082178    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"
Nov 02 13:18:28.291716 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:18:28.291679    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:18:29.693253 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:18:29.693217    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:18:55.281352 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:18:55.281315    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:18:55.325445 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:18:55.325405    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"]
Nov 02 13:18:55.342578 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:18:55.342553    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"]
Nov 02 13:18:58.291280 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:18:58.291241    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:19:00.081917 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:00.081875    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4" podUID="bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:19:07.080530 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:07.080490    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:19:10.081991 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.081943    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4" podUID="bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:19:10.349092 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.349008    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn" podUID="e41485cb-1f46-4207-9e2f-20e1f05f2502" containerName="main" containerID="cri-o://c23585430013a6d482b30891a44122a5aa6b8692ddb8bc90a334a32409f33011" gracePeriod=45
Nov 02 13:19:10.363492 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.363462    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4" podUID="bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9" containerName="main" containerID="cri-o://489a96837f404967e6539fb90a24033b836b61d371cc76884aba4aaa3684f462" gracePeriod=45
Nov 02 13:19:10.572597 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.572564    4464 generic.go:356] "Generic (PLEG): container finished" podID="e41485cb-1f46-4207-9e2f-20e1f05f2502" containerID="c23585430013a6d482b30891a44122a5aa6b8692ddb8bc90a334a32409f33011" exitCode=0
Nov 02 13:19:10.572772 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.572646    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn" event={"ID":"e41485cb-1f46-4207-9e2f-20e1f05f2502","Type":"ContainerDied","Data":"c23585430013a6d482b30891a44122a5aa6b8692ddb8bc90a334a32409f33011"}
Nov 02 13:19:10.597182 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.597158    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:19:10.710523 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.710439    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-model-cache\") pod \"e41485cb-1f46-4207-9e2f-20e1f05f2502\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") "
Nov 02 13:19:10.710523 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.710495    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-tmp-dir\") pod \"e41485cb-1f46-4207-9e2f-20e1f05f2502\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") "
Nov 02 13:19:10.710777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.710548    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41485cb-1f46-4207-9e2f-20e1f05f2502-kube-api-access-mnzdq\" (UniqueName: \"kubernetes.io/projected/e41485cb-1f46-4207-9e2f-20e1f05f2502-kube-api-access-mnzdq\") pod \"e41485cb-1f46-4207-9e2f-20e1f05f2502\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") "
Nov 02 13:19:10.710777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.710591    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e41485cb-1f46-4207-9e2f-20e1f05f2502-tls-certs\" (UniqueName: \"kubernetes.io/secret/e41485cb-1f46-4207-9e2f-20e1f05f2502-tls-certs\") pod \"e41485cb-1f46-4207-9e2f-20e1f05f2502\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") "
Nov 02 13:19:10.710777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.710650    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-home\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-home\") pod \"e41485cb-1f46-4207-9e2f-20e1f05f2502\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") "
Nov 02 13:19:10.710777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.710760    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-dshm\") pod \"e41485cb-1f46-4207-9e2f-20e1f05f2502\" (UID: \"e41485cb-1f46-4207-9e2f-20e1f05f2502\") "
Nov 02 13:19:10.710777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.710732    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-model-cache" pod "e41485cb-1f46-4207-9e2f-20e1f05f2502" (UID: "e41485cb-1f46-4207-9e2f-20e1f05f2502"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:10.711047 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.710806    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-tmp-dir" pod "e41485cb-1f46-4207-9e2f-20e1f05f2502" (UID: "e41485cb-1f46-4207-9e2f-20e1f05f2502"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:10.711047 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.710952    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-home" pod "e41485cb-1f46-4207-9e2f-20e1f05f2502" (UID: "e41485cb-1f46-4207-9e2f-20e1f05f2502"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:10.711162 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.711118    4464 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-home\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:19:10.711162 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.711133    4464 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-model-cache\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:19:10.711162 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.711142    4464 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-tmp-dir\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:19:10.712821 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.712793    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-dshm" pod "e41485cb-1f46-4207-9e2f-20e1f05f2502" (UID: "e41485cb-1f46-4207-9e2f-20e1f05f2502"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:10.712936 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.712800    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e41485cb-1f46-4207-9e2f-20e1f05f2502-tls-certs" pod "e41485cb-1f46-4207-9e2f-20e1f05f2502" (UID: "e41485cb-1f46-4207-9e2f-20e1f05f2502"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:19:10.712936 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.712797    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41485cb-1f46-4207-9e2f-20e1f05f2502-kube-api-access-mnzdq" pod "e41485cb-1f46-4207-9e2f-20e1f05f2502" (UID: "e41485cb-1f46-4207-9e2f-20e1f05f2502"). InnerVolumeSpecName "kube-api-access-mnzdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:19:10.812025 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.811965    4464 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41485cb-1f46-4207-9e2f-20e1f05f2502-dshm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:19:10.812025 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.812018    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnzdq\" (UniqueName: \"kubernetes.io/projected/e41485cb-1f46-4207-9e2f-20e1f05f2502-kube-api-access-mnzdq\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:19:10.812025 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:10.812029    4464 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41485cb-1f46-4207-9e2f-20e1f05f2502-tls-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:19:11.579882 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:11.579852    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"
Nov 02 13:19:11.581892 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:11.581862    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn" event={"ID":"e41485cb-1f46-4207-9e2f-20e1f05f2502","Type":"ContainerDied","Data":"ed31ed8fa06372a7aeb3f3a8aaabdd3127bfe83d613973c1bcfb07b1e5c8bd8e"}
Nov 02 13:19:11.582065 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:11.581897    4464 scope.go:122] "RemoveContainer" containerID="c23585430013a6d482b30891a44122a5aa6b8692ddb8bc90a334a32409f33011"
Nov 02 13:19:11.747045 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:11.747018    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"]
Nov 02 13:19:11.756039 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:11.756016    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-b2fa3bcb-kserve-7c78cd7cc4-sjtbn"]
Nov 02 13:19:13.581196 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:13.581161    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e41485cb-1f46-4207-9e2f-20e1f05f2502" path="/var/lib/kubelet/pods/e41485cb-1f46-4207-9e2f-20e1f05f2502/volumes"
Nov 02 13:19:14.668935 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:14.668909    4464 scope.go:122] "RemoveContainer" containerID="3fd88571d203d0f3febd9e02525eaff4acc4d4f8e8f40005b5b9f639ba89ac93"
Nov 02 13:19:14.677428 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:14.677411    4464 scope.go:122] "RemoveContainer" containerID="0d76458c3954a9916bad9a82f317e8418b86da057daa972bb6f025546f2de8e2"
Nov 02 13:19:14.684781 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:14.684762    4464 scope.go:122] "RemoveContainer" containerID="126bf4e54e1ac432e344604f32670f8f3c17e4ad49ca6e59f1e324f7516c7ece"
Nov 02 13:19:20.102102 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:20.102060    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4" podUID="bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:19:20.102781 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:20.102756    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"
Nov 02 13:19:28.291637 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:19:28.291599    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:19:30.082180 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:30.082142    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4" podUID="bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:19:33.351866 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:33.351829    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:19:40.081727 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:40.081687    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4" podUID="bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:19:40.617603 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:40.617572    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"
Nov 02 13:19:40.737283 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:40.737252    4464 generic.go:356] "Generic (PLEG): container finished" podID="bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9" containerID="489a96837f404967e6539fb90a24033b836b61d371cc76884aba4aaa3684f462" exitCode=0
Nov 02 13:19:40.737490 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:40.737300    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"
Nov 02 13:19:40.737490 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:40.737337    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4" event={"ID":"bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9","Type":"ContainerDied","Data":"489a96837f404967e6539fb90a24033b836b61d371cc76884aba4aaa3684f462"}
Nov 02 13:19:40.737490 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:40.737392    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4" event={"ID":"bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9","Type":"ContainerDied","Data":"f451cda5e39185451cbc9e03c7a73dddc725d7c4bec232977e2a49a4c4d162b4"}
Nov 02 13:19:40.737490 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:40.737412    4464 scope.go:122] "RemoveContainer" containerID="489a96837f404967e6539fb90a24033b836b61d371cc76884aba4aaa3684f462"
Nov 02 13:19:40.745712 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:40.745696    4464 scope.go:122] "RemoveContainer" containerID="489a96837f404967e6539fb90a24033b836b61d371cc76884aba4aaa3684f462"
Nov 02 13:19:40.746018 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:19:40.745964    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489a96837f404967e6539fb90a24033b836b61d371cc76884aba4aaa3684f462\": container with ID starting with 489a96837f404967e6539fb90a24033b836b61d371cc76884aba4aaa3684f462 not found: ID does not exist" containerID="489a96837f404967e6539fb90a24033b836b61d371cc76884aba4aaa3684f462"
Nov 02 13:19:40.746115 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:40.746027    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489a96837f404967e6539fb90a24033b836b61d371cc76884aba4aaa3684f462"} err="failed to get container status \"489a96837f404967e6539fb90a24033b836b61d371cc76884aba4aaa3684f462\": rpc error: code = NotFound desc = could not find container \"489a96837f404967e6539fb90a24033b836b61d371cc76884aba4aaa3684f462\": container with ID starting with 489a96837f404967e6539fb90a24033b836b61d371cc76884aba4aaa3684f462 not found: ID does not exist"
Nov 02 13:19:40.748468 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:40.748449    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9-tls-certs\") pod \"bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9\" (UID: \"bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9\") "
Nov 02 13:19:40.748599 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:40.748482    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9-kube-api-access-m94sh\" (UniqueName: \"kubernetes.io/projected/bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9-kube-api-access-m94sh\") pod \"bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9\" (UID: \"bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9\") "
Nov 02 13:19:40.750890 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:40.750861    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9-kube-api-access-m94sh" pod "bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9" (UID: "bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9"). InnerVolumeSpecName "kube-api-access-m94sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:19:40.751424 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:40.751401    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9-tls-certs" pod "bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9" (UID: "bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:19:40.849728 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:40.849699    4464 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9-tls-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:19:40.849728 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:40.849725    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m94sh\" (UniqueName: \"kubernetes.io/projected/bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9-kube-api-access-m94sh\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:19:41.167219 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:41.167187    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"]
Nov 02 13:19:41.217705 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:41.217674    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-b2fa3bcb-kserve-router-scheduler-757fc8b889-d6xf4"]
Nov 02 13:19:41.580866 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:41.580824    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9" path="/var/lib/kubelet/pods/bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9/volumes"
Nov 02 13:19:47.930095 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:47.930060    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"]
Nov 02 13:19:47.930507 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:47.930480    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e41485cb-1f46-4207-9e2f-20e1f05f2502" containerName="main"
Nov 02 13:19:47.930507 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:47.930491    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41485cb-1f46-4207-9e2f-20e1f05f2502" containerName="main"
Nov 02 13:19:47.930602 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:47.930530    4464 cpu_manager.go:414] "containerMap: removing container" podUID="bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9" containerName="main"
Nov 02 13:19:47.930602 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:47.930535    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9" containerName="main"
Nov 02 13:19:47.930602 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:47.930600    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdedd10e-21ed-44f6-a1d9-e755b1f0d8d9" containerName="main"
Nov 02 13:19:47.930721 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:47.930609    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41485cb-1f46-4207-9e2f-20e1f05f2502" containerName="main"
Nov 02 13:19:47.934872 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:47.934853    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:47.979333 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:47.979303    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"]
Nov 02 13:19:47.988381 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:47.988355    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-62ca10\"/\"kube-root-ca.crt\""
Nov 02 13:19:47.988518 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:47.988461    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-62ca10\"/\"default-dockercfg-g5zpg\""
Nov 02 13:19:47.988518 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:47.988472    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-62ca10\"/\"openshift-service-ca.crt\""
Nov 02 13:19:47.988617 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:47.988576    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-62ca10\"/\"simulator-single-main-62ca10-kserve-self-signed-certs\""
Nov 02 13:19:48.021030 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.021003    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh9q9\" (UniqueName: \"kubernetes.io/projected/9facf2d6-14b4-4935-a90b-5ee181593b50-kube-api-access-fh9q9\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.021140 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.021036    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-home\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.021140 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.021060    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-model-cache\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.021220 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.021156    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9facf2d6-14b4-4935-a90b-5ee181593b50-tls-certs\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.021275 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.021260    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-dshm\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.021313 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.021301    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-tmp-dir\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.122589 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.122543    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh9q9\" (UniqueName: \"kubernetes.io/projected/9facf2d6-14b4-4935-a90b-5ee181593b50-kube-api-access-fh9q9\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.122589 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.122580    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-home\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.122845 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.122607    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-model-cache\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.122845 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.122657    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9facf2d6-14b4-4935-a90b-5ee181593b50-tls-certs\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.122845 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.122760    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-dshm\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.122845 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.122807    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-tmp-dir\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.123118 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.123057    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-home\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.123118 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.123088    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-model-cache\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.123118 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.123112    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-tmp-dir\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.125001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.124969    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-dshm\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.125299 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.125279    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9facf2d6-14b4-4935-a90b-5ee181593b50-tls-certs\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.198056 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.197970    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh9q9\" (UniqueName: \"kubernetes.io/projected/9facf2d6-14b4-4935-a90b-5ee181593b50-kube-api-access-fh9q9\") pod \"simulator-single-main-62ca10-kserve-54544b6d87-4bwtf\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") " pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.258064 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.258036    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:48.633051 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.633027    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"]
Nov 02 13:19:48.634444 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:19:48.634418    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9facf2d6_14b4_4935_a90b_5ee181593b50.slice/crio-e434e734d768c01508a998d655fe6c532081bdaa60dd6b5ee2fd03d9588b9d91 WatchSource:0}: Error finding container e434e734d768c01508a998d655fe6c532081bdaa60dd6b5ee2fd03d9588b9d91: Status 404 returned error can't find the container with id e434e734d768c01508a998d655fe6c532081bdaa60dd6b5ee2fd03d9588b9d91
Nov 02 13:19:48.787832 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.787795    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf" event={"ID":"9facf2d6-14b4-4935-a90b-5ee181593b50","Type":"ContainerStarted","Data":"70fd0e240595785243d33cd68f87fdd5fb331f6a3a41f842510de29d30a740d0"}
Nov 02 13:19:48.788014 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.787840    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf" event={"ID":"9facf2d6-14b4-4935-a90b-5ee181593b50","Type":"ContainerStarted","Data":"e434e734d768c01508a998d655fe6c532081bdaa60dd6b5ee2fd03d9588b9d91"}
Nov 02 13:19:48.909558 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:48.909457    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf" podStartSLOduration=1.909443015 podStartE2EDuration="1.909443015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:19:48.90775152 +0000 UTC m=+5017.471459928" watchObservedRunningTime="2026-11-02 13:19:48.909443015 +0000 UTC m=+5017.473151424"
Nov 02 13:19:49.258680 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:49.258648    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:52.706262 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:52.706233    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:19:58.258465 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:58.258430    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:58.270740 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:58.270718    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:19:58.292284 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:19:58.292255    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:19:58.853647 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:19:58.853619    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:20:24.702556 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:24.702526    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:20:28.291493 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:20:28.291460    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:20:37.069747 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.069709    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"]
Nov 02 13:20:37.120231 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.120203    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-d8rqr"]
Nov 02 13:20:37.427388 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.427306    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"]
Nov 02 13:20:37.430904 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.430884    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"
Nov 02 13:20:37.486123 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.486087    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:20:37.536835 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.536797    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:20:37.536835 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.536815    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:20:37.536835 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.536802    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:20:37.539116 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.539092    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"]
Nov 02 13:20:37.582376 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.582353    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4de67aac-8808-4f87-ac9a-83c55cda1397" path="/var/lib/kubelet/pods/4de67aac-8808-4f87-ac9a-83c55cda1397/volumes"
Nov 02 13:20:37.616453 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.616426    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6e0e22-63ed-491d-9c58-18706c017b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7c1e6f1-27lgv\" (UID: \"cb6e0e22-63ed-491d-9c58-18706c017b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"
Nov 02 13:20:37.616572 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.616551    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2f22\" (UniqueName: \"kubernetes.io/projected/cb6e0e22-63ed-491d-9c58-18706c017b89-kube-api-access-v2f22\") pod \"maas-api-revoke-sub-e7c1e6f1-27lgv\" (UID: \"cb6e0e22-63ed-491d-9c58-18706c017b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"
Nov 02 13:20:37.616635 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.616578    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6e0e22-63ed-491d-9c58-18706c017b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7c1e6f1-27lgv\" (UID: \"cb6e0e22-63ed-491d-9c58-18706c017b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"
Nov 02 13:20:37.717907 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.717846    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6e0e22-63ed-491d-9c58-18706c017b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7c1e6f1-27lgv\" (UID: \"cb6e0e22-63ed-491d-9c58-18706c017b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"
Nov 02 13:20:37.718042 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.717921    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2f22\" (UniqueName: \"kubernetes.io/projected/cb6e0e22-63ed-491d-9c58-18706c017b89-kube-api-access-v2f22\") pod \"maas-api-revoke-sub-e7c1e6f1-27lgv\" (UID: \"cb6e0e22-63ed-491d-9c58-18706c017b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"
Nov 02 13:20:37.718042 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.717952    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6e0e22-63ed-491d-9c58-18706c017b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7c1e6f1-27lgv\" (UID: \"cb6e0e22-63ed-491d-9c58-18706c017b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"
Nov 02 13:20:37.718638 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.718611    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6e0e22-63ed-491d-9c58-18706c017b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7c1e6f1-27lgv\" (UID: \"cb6e0e22-63ed-491d-9c58-18706c017b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"
Nov 02 13:20:37.720209 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.720191    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6e0e22-63ed-491d-9c58-18706c017b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7c1e6f1-27lgv\" (UID: \"cb6e0e22-63ed-491d-9c58-18706c017b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"
Nov 02 13:20:37.780995 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:37.780959    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2f22\" (UniqueName: \"kubernetes.io/projected/cb6e0e22-63ed-491d-9c58-18706c017b89-kube-api-access-v2f22\") pod \"maas-api-revoke-sub-e7c1e6f1-27lgv\" (UID: \"cb6e0e22-63ed-491d-9c58-18706c017b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"
Nov 02 13:20:38.043657 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:38.043626    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"
Nov 02 13:20:38.414288 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:38.414260    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"]
Nov 02 13:20:38.416159 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:20:38.416133    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6e0e22_63ed_491d_9c58_18706c017b89.slice/crio-76f918c8d1b19fa7e12e78b60d410b3481f68e479022cd9c2ce38f12ceec5612 WatchSource:0}: Error finding container 76f918c8d1b19fa7e12e78b60d410b3481f68e479022cd9c2ce38f12ceec5612: Status 404 returned error can't find the container with id 76f918c8d1b19fa7e12e78b60d410b3481f68e479022cd9c2ce38f12ceec5612
Nov 02 13:20:39.043476 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:39.043439    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv" event={"ID":"cb6e0e22-63ed-491d-9c58-18706c017b89","Type":"ContainerStarted","Data":"8a281fafe22749d53fa5fad91c253866eac48077ff9dc5796eec004593344763"}
Nov 02 13:20:39.043662 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:39.043500    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv" event={"ID":"cb6e0e22-63ed-491d-9c58-18706c017b89","Type":"ContainerStarted","Data":"76f918c8d1b19fa7e12e78b60d410b3481f68e479022cd9c2ce38f12ceec5612"}
Nov 02 13:20:39.161636 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:39.161578    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv" podStartSLOduration=2.161561873 podStartE2EDuration="2.161561873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:20:39.161058027 +0000 UTC m=+5067.724766433" watchObservedRunningTime="2026-11-02 13:20:39.161561873 +0000 UTC m=+5067.725270283"
Nov 02 13:20:40.049991 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:40.049952    4464 generic.go:356] "Generic (PLEG): container finished" podID="cb6e0e22-63ed-491d-9c58-18706c017b89" containerID="8a281fafe22749d53fa5fad91c253866eac48077ff9dc5796eec004593344763" exitCode=0
Nov 02 13:20:40.050433 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:40.050025    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv" event={"ID":"cb6e0e22-63ed-491d-9c58-18706c017b89","Type":"ContainerDied","Data":"8a281fafe22749d53fa5fad91c253866eac48077ff9dc5796eec004593344763"}
Nov 02 13:20:41.184697 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:41.184672    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"
Nov 02 13:20:41.250308 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:41.250274    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb6e0e22-63ed-491d-9c58-18706c017b89-kube-api-access-v2f22\" (UniqueName: \"kubernetes.io/projected/cb6e0e22-63ed-491d-9c58-18706c017b89-kube-api-access-v2f22\") pod \"cb6e0e22-63ed-491d-9c58-18706c017b89\" (UID: \"cb6e0e22-63ed-491d-9c58-18706c017b89\") "
Nov 02 13:20:41.250517 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:41.250378    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb6e0e22-63ed-491d-9c58-18706c017b89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6e0e22-63ed-491d-9c58-18706c017b89-maas-api-xks-ca\") pod \"cb6e0e22-63ed-491d-9c58-18706c017b89\" (UID: \"cb6e0e22-63ed-491d-9c58-18706c017b89\") "
Nov 02 13:20:41.250517 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:41.250407    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb6e0e22-63ed-491d-9c58-18706c017b89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6e0e22-63ed-491d-9c58-18706c017b89-maas-api-openshift-ca\") pod \"cb6e0e22-63ed-491d-9c58-18706c017b89\" (UID: \"cb6e0e22-63ed-491d-9c58-18706c017b89\") "
Nov 02 13:20:41.251070 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:41.251037    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb6e0e22-63ed-491d-9c58-18706c017b89-maas-api-openshift-ca" pod "cb6e0e22-63ed-491d-9c58-18706c017b89" (UID: "cb6e0e22-63ed-491d-9c58-18706c017b89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:20:41.252939 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:41.252855    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6e0e22-63ed-491d-9c58-18706c017b89-kube-api-access-v2f22" pod "cb6e0e22-63ed-491d-9c58-18706c017b89" (UID: "cb6e0e22-63ed-491d-9c58-18706c017b89"). InnerVolumeSpecName "kube-api-access-v2f22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:20:41.253109 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:41.252996    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb6e0e22-63ed-491d-9c58-18706c017b89-maas-api-xks-ca" pod "cb6e0e22-63ed-491d-9c58-18706c017b89" (UID: "cb6e0e22-63ed-491d-9c58-18706c017b89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:20:41.351370 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:41.351297    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb6e0e22-63ed-491d-9c58-18706c017b89-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:20:41.351370 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:41.351327    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb6e0e22-63ed-491d-9c58-18706c017b89-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:20:41.351370 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:41.351343    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2f22\" (UniqueName: \"kubernetes.io/projected/cb6e0e22-63ed-491d-9c58-18706c017b89-kube-api-access-v2f22\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:20:42.061346 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:42.061308    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv" event={"ID":"cb6e0e22-63ed-491d-9c58-18706c017b89","Type":"ContainerDied","Data":"76f918c8d1b19fa7e12e78b60d410b3481f68e479022cd9c2ce38f12ceec5612"}
Nov 02 13:20:42.061562 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:42.061369    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76f918c8d1b19fa7e12e78b60d410b3481f68e479022cd9c2ce38f12ceec5612"
Nov 02 13:20:42.061562 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:42.061323    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"
Nov 02 13:20:50.411114 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:50.411073    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:20:53.538306 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:20:53.538270    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"]
Nov 02 13:20:58.291015 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:20:58.290921    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:21:00.509383 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:00.509349    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:21:08.557816 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:08.557777    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf" podUID="9facf2d6-14b4-4935-a90b-5ee181593b50" containerName="main" containerID="cri-o://70fd0e240595785243d33cd68f87fdd5fb331f6a3a41f842510de29d30a740d0" gracePeriod=45
Nov 02 13:21:08.804565 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:08.804543    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:21:08.909599 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:08.909544    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-tmp-dir\") pod \"9facf2d6-14b4-4935-a90b-5ee181593b50\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") "
Nov 02 13:21:08.909599 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:08.909582    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-home\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-home\") pod \"9facf2d6-14b4-4935-a90b-5ee181593b50\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") "
Nov 02 13:21:08.909888 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:08.909621    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9facf2d6-14b4-4935-a90b-5ee181593b50-kube-api-access-fh9q9\" (UniqueName: \"kubernetes.io/projected/9facf2d6-14b4-4935-a90b-5ee181593b50-kube-api-access-fh9q9\") pod \"9facf2d6-14b4-4935-a90b-5ee181593b50\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") "
Nov 02 13:21:08.909888 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:08.909709    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-dshm\") pod \"9facf2d6-14b4-4935-a90b-5ee181593b50\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") "
Nov 02 13:21:08.909888 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:08.909754    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-model-cache\") pod \"9facf2d6-14b4-4935-a90b-5ee181593b50\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") "
Nov 02 13:21:08.909888 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:08.909808    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9facf2d6-14b4-4935-a90b-5ee181593b50-tls-certs\" (UniqueName: \"kubernetes.io/secret/9facf2d6-14b4-4935-a90b-5ee181593b50-tls-certs\") pod \"9facf2d6-14b4-4935-a90b-5ee181593b50\" (UID: \"9facf2d6-14b4-4935-a90b-5ee181593b50\") "
Nov 02 13:21:08.909888 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:08.909822    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-home" pod "9facf2d6-14b4-4935-a90b-5ee181593b50" (UID: "9facf2d6-14b4-4935-a90b-5ee181593b50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:08.909888 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:08.909826    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-tmp-dir" pod "9facf2d6-14b4-4935-a90b-5ee181593b50" (UID: "9facf2d6-14b4-4935-a90b-5ee181593b50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:08.910218 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:08.910040    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-model-cache" pod "9facf2d6-14b4-4935-a90b-5ee181593b50" (UID: "9facf2d6-14b4-4935-a90b-5ee181593b50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:08.910218 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:08.910157    4464 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-tmp-dir\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:21:08.910218 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:08.910172    4464 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-home\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:21:08.910218 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:08.910180    4464 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-model-cache\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:21:08.912006 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:08.911967    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9facf2d6-14b4-4935-a90b-5ee181593b50-tls-certs" pod "9facf2d6-14b4-4935-a90b-5ee181593b50" (UID: "9facf2d6-14b4-4935-a90b-5ee181593b50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:21:08.912119 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:08.912061    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9facf2d6-14b4-4935-a90b-5ee181593b50-kube-api-access-fh9q9" pod "9facf2d6-14b4-4935-a90b-5ee181593b50" (UID: "9facf2d6-14b4-4935-a90b-5ee181593b50"). InnerVolumeSpecName "kube-api-access-fh9q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:21:08.912205 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:08.912185    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-dshm" pod "9facf2d6-14b4-4935-a90b-5ee181593b50" (UID: "9facf2d6-14b4-4935-a90b-5ee181593b50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:09.011464 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:09.011429    4464 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9facf2d6-14b4-4935-a90b-5ee181593b50-dshm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:21:09.011464 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:09.011461    4464 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9facf2d6-14b4-4935-a90b-5ee181593b50-tls-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:21:09.011671 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:09.011475    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh9q9\" (UniqueName: \"kubernetes.io/projected/9facf2d6-14b4-4935-a90b-5ee181593b50-kube-api-access-fh9q9\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:21:09.207582 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:09.207499    4464 generic.go:356] "Generic (PLEG): container finished" podID="9facf2d6-14b4-4935-a90b-5ee181593b50" containerID="70fd0e240595785243d33cd68f87fdd5fb331f6a3a41f842510de29d30a740d0" exitCode=0
Nov 02 13:21:09.207582 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:09.207547    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"
Nov 02 13:21:09.207791 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:09.207583    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf" event={"ID":"9facf2d6-14b4-4935-a90b-5ee181593b50","Type":"ContainerDied","Data":"70fd0e240595785243d33cd68f87fdd5fb331f6a3a41f842510de29d30a740d0"}
Nov 02 13:21:09.207791 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:09.207627    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf" event={"ID":"9facf2d6-14b4-4935-a90b-5ee181593b50","Type":"ContainerDied","Data":"e434e734d768c01508a998d655fe6c532081bdaa60dd6b5ee2fd03d9588b9d91"}
Nov 02 13:21:09.207791 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:09.207645    4464 scope.go:122] "RemoveContainer" containerID="70fd0e240595785243d33cd68f87fdd5fb331f6a3a41f842510de29d30a740d0"
Nov 02 13:21:09.217174 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:09.217160    4464 scope.go:122] "RemoveContainer" containerID="70fd0e240595785243d33cd68f87fdd5fb331f6a3a41f842510de29d30a740d0"
Nov 02 13:21:09.217439 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:21:09.217423    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70fd0e240595785243d33cd68f87fdd5fb331f6a3a41f842510de29d30a740d0\": container with ID starting with 70fd0e240595785243d33cd68f87fdd5fb331f6a3a41f842510de29d30a740d0 not found: ID does not exist" containerID="70fd0e240595785243d33cd68f87fdd5fb331f6a3a41f842510de29d30a740d0"
Nov 02 13:21:09.217493 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:09.217445    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70fd0e240595785243d33cd68f87fdd5fb331f6a3a41f842510de29d30a740d0"} err="failed to get container status \"70fd0e240595785243d33cd68f87fdd5fb331f6a3a41f842510de29d30a740d0\": rpc error: code = NotFound desc = could not find container \"70fd0e240595785243d33cd68f87fdd5fb331f6a3a41f842510de29d30a740d0\": container with ID starting with 70fd0e240595785243d33cd68f87fdd5fb331f6a3a41f842510de29d30a740d0 not found: ID does not exist"
Nov 02 13:21:09.375255 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:09.375222    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"]
Nov 02 13:21:09.378875 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:09.378853    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-62ca10/simulator-single-main-62ca10-kserve-54544b6d87-4bwtf"]
Nov 02 13:21:09.581643 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:09.581614    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9facf2d6-14b4-4935-a90b-5ee181593b50" path="/var/lib/kubelet/pods/9facf2d6-14b4-4935-a90b-5ee181593b50/volumes"
Nov 02 13:21:12.240862 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:12.240832    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:21:12.256578 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:12.256555    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:21:12.278230 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:12.278210    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:21:12.293735 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:12.293711    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:21:14.730831 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:14.730802    4464 scope.go:122] "RemoveContainer" containerID="518c0dd96f63ee881a567312b0f43f96428175b3d9a9f5053be2a4352fab5ca6"
Nov 02 13:21:28.291104 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:21:28.291070    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:21:41.502200 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:41.502165    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:21:44.425119 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.425086    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"]
Nov 02 13:21:44.425607 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.425515    4464 cpu_manager.go:414] "containerMap: removing container" podUID="cb6e0e22-63ed-491d-9c58-18706c017b89" containerName="revoke-keys"
Nov 02 13:21:44.425607 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.425526    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb6e0e22-63ed-491d-9c58-18706c017b89" containerName="revoke-keys"
Nov 02 13:21:44.425607 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.425540    4464 cpu_manager.go:414] "containerMap: removing container" podUID="9facf2d6-14b4-4935-a90b-5ee181593b50" containerName="main"
Nov 02 13:21:44.425607 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.425545    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9facf2d6-14b4-4935-a90b-5ee181593b50" containerName="main"
Nov 02 13:21:44.425827 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.425624    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb6e0e22-63ed-491d-9c58-18706c017b89" containerName="revoke-keys"
Nov 02 13:21:44.425827 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.425632    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="9facf2d6-14b4-4935-a90b-5ee181593b50" containerName="main"
Nov 02 13:21:44.428738 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.428721    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"
Nov 02 13:21:44.535125 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.535096    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:21:44.535125 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.535113    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:21:44.535343 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.535145    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:21:44.535343 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.535172    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:21:44.535516 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.535493    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ef3826-95fe-414a-ab47-dd2a983ebab5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9\" (UID: \"25ef3826-95fe-414a-ab47-dd2a983ebab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"
Nov 02 13:21:44.535595 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.535578    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ef3826-95fe-414a-ab47-dd2a983ebab5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9\" (UID: \"25ef3826-95fe-414a-ab47-dd2a983ebab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"
Nov 02 13:21:44.535686 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.535672    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vj97\" (UniqueName: \"kubernetes.io/projected/25ef3826-95fe-414a-ab47-dd2a983ebab5-kube-api-access-9vj97\") pod \"maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9\" (UID: \"25ef3826-95fe-414a-ab47-dd2a983ebab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"
Nov 02 13:21:44.537398 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.537377    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"]
Nov 02 13:21:44.636672 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.636647    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ef3826-95fe-414a-ab47-dd2a983ebab5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9\" (UID: \"25ef3826-95fe-414a-ab47-dd2a983ebab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"
Nov 02 13:21:44.636842 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.636686    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ef3826-95fe-414a-ab47-dd2a983ebab5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9\" (UID: \"25ef3826-95fe-414a-ab47-dd2a983ebab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"
Nov 02 13:21:44.636842 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.636746    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vj97\" (UniqueName: \"kubernetes.io/projected/25ef3826-95fe-414a-ab47-dd2a983ebab5-kube-api-access-9vj97\") pod \"maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9\" (UID: \"25ef3826-95fe-414a-ab47-dd2a983ebab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"
Nov 02 13:21:44.637406 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.637372    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ef3826-95fe-414a-ab47-dd2a983ebab5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9\" (UID: \"25ef3826-95fe-414a-ab47-dd2a983ebab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"
Nov 02 13:21:44.639071 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.639046    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ef3826-95fe-414a-ab47-dd2a983ebab5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9\" (UID: \"25ef3826-95fe-414a-ab47-dd2a983ebab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"
Nov 02 13:21:44.698715 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.698654    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vj97\" (UniqueName: \"kubernetes.io/projected/25ef3826-95fe-414a-ab47-dd2a983ebab5-kube-api-access-9vj97\") pod \"maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9\" (UID: \"25ef3826-95fe-414a-ab47-dd2a983ebab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"
Nov 02 13:21:44.742457 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:44.742433    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"
Nov 02 13:21:45.116740 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:45.116715    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"]
Nov 02 13:21:45.118325 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:21:45.118297    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ef3826_95fe_414a_ab47_dd2a983ebab5.slice/crio-168a8331ee195548dbd34caf084e5d6151a3e361bb259affc41c2c686594afcd WatchSource:0}: Error finding container 168a8331ee195548dbd34caf084e5d6151a3e361bb259affc41c2c686594afcd: Status 404 returned error can't find the container with id 168a8331ee195548dbd34caf084e5d6151a3e361bb259affc41c2c686594afcd
Nov 02 13:21:45.412671 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:45.412635    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9" event={"ID":"25ef3826-95fe-414a-ab47-dd2a983ebab5","Type":"ContainerStarted","Data":"ff7b7fa14b813230cb4bde90efbdf77c9501392465938e3477f37427462e5d59"}
Nov 02 13:21:45.412789 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:45.412686    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9" event={"ID":"25ef3826-95fe-414a-ab47-dd2a983ebab5","Type":"ContainerStarted","Data":"168a8331ee195548dbd34caf084e5d6151a3e361bb259affc41c2c686594afcd"}
Nov 02 13:21:45.530294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:45.530244    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9" podStartSLOduration=1.5302296050000002 podStartE2EDuration="1.530229605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:21:45.529632968 +0000 UTC m=+5134.093341376" watchObservedRunningTime="2026-11-02 13:21:45.530229605 +0000 UTC m=+5134.093938036"
Nov 02 13:21:46.418480 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:46.418391    4464 generic.go:356] "Generic (PLEG): container finished" podID="25ef3826-95fe-414a-ab47-dd2a983ebab5" containerID="ff7b7fa14b813230cb4bde90efbdf77c9501392465938e3477f37427462e5d59" exitCode=0
Nov 02 13:21:46.418642 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:46.418474    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9" event={"ID":"25ef3826-95fe-414a-ab47-dd2a983ebab5","Type":"ContainerDied","Data":"ff7b7fa14b813230cb4bde90efbdf77c9501392465938e3477f37427462e5d59"}
Nov 02 13:21:47.559139 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:47.559112    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"
Nov 02 13:21:47.665719 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:47.665684    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25ef3826-95fe-414a-ab47-dd2a983ebab5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ef3826-95fe-414a-ab47-dd2a983ebab5-maas-api-xks-ca\") pod \"25ef3826-95fe-414a-ab47-dd2a983ebab5\" (UID: \"25ef3826-95fe-414a-ab47-dd2a983ebab5\") "
Nov 02 13:21:47.665892 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:47.665763    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ef3826-95fe-414a-ab47-dd2a983ebab5-kube-api-access-9vj97\" (UniqueName: \"kubernetes.io/projected/25ef3826-95fe-414a-ab47-dd2a983ebab5-kube-api-access-9vj97\") pod \"25ef3826-95fe-414a-ab47-dd2a983ebab5\" (UID: \"25ef3826-95fe-414a-ab47-dd2a983ebab5\") "
Nov 02 13:21:47.665892 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:47.665810    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ef3826-95fe-414a-ab47-dd2a983ebab5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ef3826-95fe-414a-ab47-dd2a983ebab5-maas-api-openshift-ca\") pod \"25ef3826-95fe-414a-ab47-dd2a983ebab5\" (UID: \"25ef3826-95fe-414a-ab47-dd2a983ebab5\") "
Nov 02 13:21:47.666371 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:47.666239    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ef3826-95fe-414a-ab47-dd2a983ebab5-maas-api-openshift-ca" pod "25ef3826-95fe-414a-ab47-dd2a983ebab5" (UID: "25ef3826-95fe-414a-ab47-dd2a983ebab5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:21:47.667940 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:47.667911    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ef3826-95fe-414a-ab47-dd2a983ebab5-maas-api-xks-ca" pod "25ef3826-95fe-414a-ab47-dd2a983ebab5" (UID: "25ef3826-95fe-414a-ab47-dd2a983ebab5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:21:47.668076 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:47.667958    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ef3826-95fe-414a-ab47-dd2a983ebab5-kube-api-access-9vj97" pod "25ef3826-95fe-414a-ab47-dd2a983ebab5" (UID: "25ef3826-95fe-414a-ab47-dd2a983ebab5"). InnerVolumeSpecName "kube-api-access-9vj97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:21:47.767116 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:47.767090    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ef3826-95fe-414a-ab47-dd2a983ebab5-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:21:47.767116 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:47.767113    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ef3826-95fe-414a-ab47-dd2a983ebab5-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:21:47.767321 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:47.767125    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vj97\" (UniqueName: \"kubernetes.io/projected/25ef3826-95fe-414a-ab47-dd2a983ebab5-kube-api-access-9vj97\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:21:48.430416 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:48.430380    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9" event={"ID":"25ef3826-95fe-414a-ab47-dd2a983ebab5","Type":"ContainerDied","Data":"168a8331ee195548dbd34caf084e5d6151a3e361bb259affc41c2c686594afcd"}
Nov 02 13:21:48.430416 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:48.430408    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"
Nov 02 13:21:48.430654 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:21:48.430430    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="168a8331ee195548dbd34caf084e5d6151a3e361bb259affc41c2c686594afcd"
Nov 02 13:21:58.291040 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:21:58.291003    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:22:01.048307 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:01.048269    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"]
Nov 02 13:22:01.098631 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:01.098594    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-h4z29"]
Nov 02 13:22:01.580794 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:01.580751    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2634b75b-1e7c-46f1-9f54-437fdd72dd8c" path="/var/lib/kubelet/pods/2634b75b-1e7c-46f1-9f54-437fdd72dd8c/volumes"
Nov 02 13:22:03.837267 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:03.837237    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"]
Nov 02 13:22:03.887832 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:03.887803    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-62ca10-xjxr9"]
Nov 02 13:22:05.581438 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:05.581403    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25ef3826-95fe-414a-ab47-dd2a983ebab5" path="/var/lib/kubelet/pods/25ef3826-95fe-414a-ab47-dd2a983ebab5/volumes"
Nov 02 13:22:07.789401 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:07.789365    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:22:14.754719 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:14.754685    4464 scope.go:122] "RemoveContainer" containerID="52f3ae959275858316282b82459f7537e2e185d655d3cfa3f2778cb9466f2920"
Nov 02 13:22:22.707661 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:22.707628    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:22:23.047971 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:23.047936    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"]
Nov 02 13:22:23.098234 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:23.098202    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7sq46"]
Nov 02 13:22:23.581041 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:23.581007    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45" path="/var/lib/kubelet/pods/0ac77eb8-8a43-430a-9ffd-6ad3a96d6c45/volumes"
Nov 02 13:22:28.291186 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:22:28.291104    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:22:35.055430 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:35.055398    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"]
Nov 02 13:22:35.105892 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:35.105860    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9m49s"]
Nov 02 13:22:35.581675 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:35.581643    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b19b839-2de9-48f4-9877-6d823d442d2b" path="/var/lib/kubelet/pods/8b19b839-2de9-48f4-9877-6d823d442d2b/volumes"
Nov 02 13:22:37.818277 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.818245    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"]
Nov 02 13:22:37.818684 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.818669    4464 cpu_manager.go:414] "containerMap: removing container" podUID="25ef3826-95fe-414a-ab47-dd2a983ebab5" containerName="revoke-keys"
Nov 02 13:22:37.818733 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.818685    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ef3826-95fe-414a-ab47-dd2a983ebab5" containerName="revoke-keys"
Nov 02 13:22:37.818770 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.818765    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ef3826-95fe-414a-ab47-dd2a983ebab5" containerName="revoke-keys"
Nov 02 13:22:37.821956 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.821936    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:37.888131 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.888102    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tn47\" (UniqueName: \"kubernetes.io/projected/56a940cb-a4a0-4593-a5a7-04883f9cab24-kube-api-access-6tn47\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:37.888321 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.888141    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-tmp-dir\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:37.888321 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.888248    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-home\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:37.888321 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.888291    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a940cb-a4a0-4593-a5a7-04883f9cab24-tls-certs\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:37.888473 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.888346    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-model-cache\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:37.888473 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.888389    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-dshm\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:37.926228 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.926198    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-128a019f-kserve-self-signed-certs\""
Nov 02 13:22:37.928229 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.928208    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"]
Nov 02 13:22:37.989679 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.989647    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tn47\" (UniqueName: \"kubernetes.io/projected/56a940cb-a4a0-4593-a5a7-04883f9cab24-kube-api-access-6tn47\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:37.989861 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.989687    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-tmp-dir\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:37.989861 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.989760    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-home\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:37.989861 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.989785    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a940cb-a4a0-4593-a5a7-04883f9cab24-tls-certs\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:37.989861 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.989808    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-model-cache\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:37.990080 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.989872    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-dshm\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:37.990232 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.990202    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-tmp-dir\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:37.990322 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.990231    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-home\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:37.990322 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.990281    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-model-cache\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:37.992149 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.992126    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-dshm\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:37.992338 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:37.992318    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a940cb-a4a0-4593-a5a7-04883f9cab24-tls-certs\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:38.052141 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:38.052111    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tn47\" (UniqueName: \"kubernetes.io/projected/56a940cb-a4a0-4593-a5a7-04883f9cab24-kube-api-access-6tn47\") pod \"e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") " pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:38.145809 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:38.145742    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:38.546206 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:38.546176    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"]
Nov 02 13:22:38.547751 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:22:38.547718    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56a940cb_a4a0_4593_a5a7_04883f9cab24.slice/crio-f8250ac791eda078fa6392dec526b8470e4a30e085175adddcc1266d6006bc3d WatchSource:0}: Error finding container f8250ac791eda078fa6392dec526b8470e4a30e085175adddcc1266d6006bc3d: Status 404 returned error can't find the container with id f8250ac791eda078fa6392dec526b8470e4a30e085175adddcc1266d6006bc3d
Nov 02 13:22:38.730079 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:38.730042    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c" event={"ID":"56a940cb-a4a0-4593-a5a7-04883f9cab24","Type":"ContainerStarted","Data":"19daf8bc10a6559d7f2c84b7d613751f94694cd2d43d93c902f652d2a01cda59"}
Nov 02 13:22:38.730260 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:38.730090    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c" event={"ID":"56a940cb-a4a0-4593-a5a7-04883f9cab24","Type":"ContainerStarted","Data":"f8250ac791eda078fa6392dec526b8470e4a30e085175adddcc1266d6006bc3d"}
Nov 02 13:22:38.850166 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:38.850069    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c" podStartSLOduration=1.850056304 podStartE2EDuration="1.850056304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:22:38.849183174 +0000 UTC m=+5187.412891585" watchObservedRunningTime="2026-11-02 13:22:38.850056304 +0000 UTC m=+5187.413764713"
Nov 02 13:22:39.146182 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:39.146103    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:42.665400 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:42.665360    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:22:48.146887 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:48.146853    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:48.159658 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:48.159633    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:48.818140 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:22:48.818110    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:22:58.290818 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:22:58.290782    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:23:07.523062 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:07.523028    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"]
Nov 02 13:23:07.526502 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:07.526486    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"
Nov 02 13:23:07.585090 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:07.585064    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"]
Nov 02 13:23:07.626620 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:07.626587    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4wc2m\" (UID: \"70d7e5d5-5bb2-4eca-867f-f63a088fdd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"
Nov 02 13:23:07.626759 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:07.626633    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4wc2m\" (UID: \"70d7e5d5-5bb2-4eca-867f-f63a088fdd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"
Nov 02 13:23:07.626759 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:07.626749    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbvdb\" (UniqueName: \"kubernetes.io/projected/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-kube-api-access-dbvdb\") pod \"maas-api-revoke-sub-452b6d7b-4wc2m\" (UID: \"70d7e5d5-5bb2-4eca-867f-f63a088fdd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"
Nov 02 13:23:07.632638 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:07.632613    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:23:07.632892 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:07.632876    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:23:07.632950 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:07.632883    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:23:07.633142 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:07.633127    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:23:07.727316 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:07.727292    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4wc2m\" (UID: \"70d7e5d5-5bb2-4eca-867f-f63a088fdd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"
Nov 02 13:23:07.727461 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:07.727328    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4wc2m\" (UID: \"70d7e5d5-5bb2-4eca-867f-f63a088fdd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"
Nov 02 13:23:07.727461 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:07.727405    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbvdb\" (UniqueName: \"kubernetes.io/projected/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-kube-api-access-dbvdb\") pod \"maas-api-revoke-sub-452b6d7b-4wc2m\" (UID: \"70d7e5d5-5bb2-4eca-867f-f63a088fdd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"
Nov 02 13:23:07.727929 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:07.727906    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4wc2m\" (UID: \"70d7e5d5-5bb2-4eca-867f-f63a088fdd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"
Nov 02 13:23:07.729673 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:07.729650    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4wc2m\" (UID: \"70d7e5d5-5bb2-4eca-867f-f63a088fdd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"
Nov 02 13:23:07.790195 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:07.790142    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbvdb\" (UniqueName: \"kubernetes.io/projected/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-kube-api-access-dbvdb\") pod \"maas-api-revoke-sub-452b6d7b-4wc2m\" (UID: \"70d7e5d5-5bb2-4eca-867f-f63a088fdd82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"
Nov 02 13:23:07.839377 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:07.839355    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"
Nov 02 13:23:08.210960 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:08.210931    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"]
Nov 02 13:23:08.212090 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:23:08.212061    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d7e5d5_5bb2_4eca_867f_f63a088fdd82.slice/crio-c80eba4d7c8fafdbeefc260cefa6242240b7e18687ff3c8295c1f4c40afc0750 WatchSource:0}: Error finding container c80eba4d7c8fafdbeefc260cefa6242240b7e18687ff3c8295c1f4c40afc0750: Status 404 returned error can't find the container with id c80eba4d7c8fafdbeefc260cefa6242240b7e18687ff3c8295c1f4c40afc0750
Nov 02 13:23:08.910119 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:08.910084    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m" event={"ID":"70d7e5d5-5bb2-4eca-867f-f63a088fdd82","Type":"ContainerStarted","Data":"ddaa213feeae439912c593fbf71eab295f4424e906946c71cfc8b4ad349dff64"}
Nov 02 13:23:08.910498 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:08.910138    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m" event={"ID":"70d7e5d5-5bb2-4eca-867f-f63a088fdd82","Type":"ContainerStarted","Data":"c80eba4d7c8fafdbeefc260cefa6242240b7e18687ff3c8295c1f4c40afc0750"}
Nov 02 13:23:09.018246 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:09.018212    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:23:09.050730 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:09.050684    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m" podStartSLOduration=2.050668441 podStartE2EDuration="2.050668441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:23:09.048781185 +0000 UTC m=+5217.612489593" watchObservedRunningTime="2026-11-02 13:23:09.050668441 +0000 UTC m=+5217.614376851"
Nov 02 13:23:09.918664 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:09.918626    4464 generic.go:356] "Generic (PLEG): container finished" podID="70d7e5d5-5bb2-4eca-867f-f63a088fdd82" containerID="ddaa213feeae439912c593fbf71eab295f4424e906946c71cfc8b4ad349dff64" exitCode=0
Nov 02 13:23:09.919110 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:09.918717    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m" event={"ID":"70d7e5d5-5bb2-4eca-867f-f63a088fdd82","Type":"ContainerDied","Data":"ddaa213feeae439912c593fbf71eab295f4424e906946c71cfc8b4ad349dff64"}
Nov 02 13:23:11.354679 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:11.354487    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"
Nov 02 13:23:11.438335 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:11.438296    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-maas-api-xks-ca\") pod \"70d7e5d5-5bb2-4eca-867f-f63a088fdd82\" (UID: \"70d7e5d5-5bb2-4eca-867f-f63a088fdd82\") "
Nov 02 13:23:11.442023 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:11.440755    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-maas-api-xks-ca" pod "70d7e5d5-5bb2-4eca-867f-f63a088fdd82" (UID: "70d7e5d5-5bb2-4eca-867f-f63a088fdd82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:23:11.541411 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:11.541347    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-kube-api-access-dbvdb\" (UniqueName: \"kubernetes.io/projected/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-kube-api-access-dbvdb\") pod \"70d7e5d5-5bb2-4eca-867f-f63a088fdd82\" (UID: \"70d7e5d5-5bb2-4eca-867f-f63a088fdd82\") "
Nov 02 13:23:11.541411 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:11.541420    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-maas-api-openshift-ca\") pod \"70d7e5d5-5bb2-4eca-867f-f63a088fdd82\" (UID: \"70d7e5d5-5bb2-4eca-867f-f63a088fdd82\") "
Nov 02 13:23:11.542516 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:11.541847    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:23:11.542516 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:11.542248    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-maas-api-openshift-ca" pod "70d7e5d5-5bb2-4eca-867f-f63a088fdd82" (UID: "70d7e5d5-5bb2-4eca-867f-f63a088fdd82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:23:11.544556 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:11.544531    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-kube-api-access-dbvdb" pod "70d7e5d5-5bb2-4eca-867f-f63a088fdd82" (UID: "70d7e5d5-5bb2-4eca-867f-f63a088fdd82"). InnerVolumeSpecName "kube-api-access-dbvdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:23:11.643121 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:11.643042    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbvdb\" (UniqueName: \"kubernetes.io/projected/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-kube-api-access-dbvdb\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:23:11.643121 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:11.643072    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d7e5d5-5bb2-4eca-867f-f63a088fdd82-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:23:11.947200 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:11.947120    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"
Nov 02 13:23:11.947353 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:11.947112    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m" event={"ID":"70d7e5d5-5bb2-4eca-867f-f63a088fdd82","Type":"ContainerDied","Data":"c80eba4d7c8fafdbeefc260cefa6242240b7e18687ff3c8295c1f4c40afc0750"}
Nov 02 13:23:11.947353 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:11.947256    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c80eba4d7c8fafdbeefc260cefa6242240b7e18687ff3c8295c1f4c40afc0750"
Nov 02 13:23:14.775796 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:14.775767    4464 scope.go:122] "RemoveContainer" containerID="3d871723a4fc6992cb6cc33cd35892ce13cfe273e94ca4e84d1b66d23bfde032"
Nov 02 13:23:24.902375 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:24.902344    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:23:28.351581 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:23:28.351544    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:23:39.090900 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:39.090864    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"]
Nov 02 13:23:43.188304 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.188269    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"]
Nov 02 13:23:43.189046 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.189023    4464 cpu_manager.go:414] "containerMap: removing container" podUID="70d7e5d5-5bb2-4eca-867f-f63a088fdd82" containerName="revoke-keys"
Nov 02 13:23:43.189122 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.189049    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70d7e5d5-5bb2-4eca-867f-f63a088fdd82" containerName="revoke-keys"
Nov 02 13:23:43.189206 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.189195    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="70d7e5d5-5bb2-4eca-867f-f63a088fdd82" containerName="revoke-keys"
Nov 02 13:23:43.192805 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.192787    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"
Nov 02 13:23:43.274029 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.273971    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7901efe8-9ffe-4931-8185-3112b79085a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xb45d\" (UID: \"7901efe8-9ffe-4931-8185-3112b79085a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"
Nov 02 13:23:43.274188 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.274120    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb874\" (UniqueName: \"kubernetes.io/projected/7901efe8-9ffe-4931-8185-3112b79085a0-kube-api-access-qb874\") pod \"maas-api-revoke-sub-3e31a506-xb45d\" (UID: \"7901efe8-9ffe-4931-8185-3112b79085a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"
Nov 02 13:23:43.274188 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.274176    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7901efe8-9ffe-4931-8185-3112b79085a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xb45d\" (UID: \"7901efe8-9ffe-4931-8185-3112b79085a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"
Nov 02 13:23:43.296907 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.296885    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:23:43.297083 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.296915    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:23:43.297083 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.296932    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:23:43.297083 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.296942    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:23:43.298957 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.298943    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"]
Nov 02 13:23:43.375133 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.375112    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7901efe8-9ffe-4931-8185-3112b79085a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xb45d\" (UID: \"7901efe8-9ffe-4931-8185-3112b79085a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"
Nov 02 13:23:43.375244 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.375175    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb874\" (UniqueName: \"kubernetes.io/projected/7901efe8-9ffe-4931-8185-3112b79085a0-kube-api-access-qb874\") pod \"maas-api-revoke-sub-3e31a506-xb45d\" (UID: \"7901efe8-9ffe-4931-8185-3112b79085a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"
Nov 02 13:23:43.375244 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.375203    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7901efe8-9ffe-4931-8185-3112b79085a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xb45d\" (UID: \"7901efe8-9ffe-4931-8185-3112b79085a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"
Nov 02 13:23:43.375717 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.375699    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7901efe8-9ffe-4931-8185-3112b79085a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-xb45d\" (UID: \"7901efe8-9ffe-4931-8185-3112b79085a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"
Nov 02 13:23:43.377465 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.377445    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7901efe8-9ffe-4931-8185-3112b79085a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-xb45d\" (UID: \"7901efe8-9ffe-4931-8185-3112b79085a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"
Nov 02 13:23:43.437135 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.437116    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb874\" (UniqueName: \"kubernetes.io/projected/7901efe8-9ffe-4931-8185-3112b79085a0-kube-api-access-qb874\") pod \"maas-api-revoke-sub-3e31a506-xb45d\" (UID: \"7901efe8-9ffe-4931-8185-3112b79085a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"
Nov 02 13:23:43.504957 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.504930    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:23:43.505260 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.505243    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"
Nov 02 13:23:43.878412 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:43.878385    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"]
Nov 02 13:23:43.880559 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:23:43.880533    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7901efe8_9ffe_4931_8185_3112b79085a0.slice/crio-9949c5c9d1e0b6e95ea2a262d95bffcc8a9f29156912db140fe6407b746bd283 WatchSource:0}: Error finding container 9949c5c9d1e0b6e95ea2a262d95bffcc8a9f29156912db140fe6407b746bd283: Status 404 returned error can't find the container with id 9949c5c9d1e0b6e95ea2a262d95bffcc8a9f29156912db140fe6407b746bd283
Nov 02 13:23:44.118871 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:44.118831    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d" event={"ID":"7901efe8-9ffe-4931-8185-3112b79085a0","Type":"ContainerStarted","Data":"9949c5c9d1e0b6e95ea2a262d95bffcc8a9f29156912db140fe6407b746bd283"}
Nov 02 13:23:45.151001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:45.150215    4464 generic.go:356] "Generic (PLEG): container finished" podID="7901efe8-9ffe-4931-8185-3112b79085a0" containerID="ab1ead3c69d9567749be2b388d4819f237a61f1e1e069cd247f411c7cd1e4634" exitCode=0
Nov 02 13:23:45.151001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:45.150413    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d" event={"ID":"7901efe8-9ffe-4931-8185-3112b79085a0","Type":"ContainerDied","Data":"ab1ead3c69d9567749be2b388d4819f237a61f1e1e069cd247f411c7cd1e4634"}
Nov 02 13:23:46.570013 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:46.567316    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"
Nov 02 13:23:46.648565 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:46.648536    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7901efe8-9ffe-4931-8185-3112b79085a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7901efe8-9ffe-4931-8185-3112b79085a0-maas-api-openshift-ca\") pod \"7901efe8-9ffe-4931-8185-3112b79085a0\" (UID: \"7901efe8-9ffe-4931-8185-3112b79085a0\") "
Nov 02 13:23:46.648757 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:46.648597    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7901efe8-9ffe-4931-8185-3112b79085a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7901efe8-9ffe-4931-8185-3112b79085a0-maas-api-xks-ca\") pod \"7901efe8-9ffe-4931-8185-3112b79085a0\" (UID: \"7901efe8-9ffe-4931-8185-3112b79085a0\") "
Nov 02 13:23:46.648757 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:46.648626    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7901efe8-9ffe-4931-8185-3112b79085a0-kube-api-access-qb874\" (UniqueName: \"kubernetes.io/projected/7901efe8-9ffe-4931-8185-3112b79085a0-kube-api-access-qb874\") pod \"7901efe8-9ffe-4931-8185-3112b79085a0\" (UID: \"7901efe8-9ffe-4931-8185-3112b79085a0\") "
Nov 02 13:23:46.648989 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:46.648950    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7901efe8-9ffe-4931-8185-3112b79085a0-maas-api-openshift-ca" pod "7901efe8-9ffe-4931-8185-3112b79085a0" (UID: "7901efe8-9ffe-4931-8185-3112b79085a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:23:46.650725 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:46.650698    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7901efe8-9ffe-4931-8185-3112b79085a0-maas-api-xks-ca" pod "7901efe8-9ffe-4931-8185-3112b79085a0" (UID: "7901efe8-9ffe-4931-8185-3112b79085a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:23:46.650832 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:46.650755    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7901efe8-9ffe-4931-8185-3112b79085a0-kube-api-access-qb874" pod "7901efe8-9ffe-4931-8185-3112b79085a0" (UID: "7901efe8-9ffe-4931-8185-3112b79085a0"). InnerVolumeSpecName "kube-api-access-qb874". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:23:46.749790 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:46.749757    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7901efe8-9ffe-4931-8185-3112b79085a0-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:23:46.749790 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:46.749786    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7901efe8-9ffe-4931-8185-3112b79085a0-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:23:46.750039 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:46.749799    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb874\" (UniqueName: \"kubernetes.io/projected/7901efe8-9ffe-4931-8185-3112b79085a0-kube-api-access-qb874\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:23:47.196999 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:47.196494    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d" event={"ID":"7901efe8-9ffe-4931-8185-3112b79085a0","Type":"ContainerDied","Data":"9949c5c9d1e0b6e95ea2a262d95bffcc8a9f29156912db140fe6407b746bd283"}
Nov 02 13:23:47.196999 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:47.196596    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9949c5c9d1e0b6e95ea2a262d95bffcc8a9f29156912db140fe6407b746bd283"
Nov 02 13:23:47.196999 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:47.196673    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"
Nov 02 13:23:54.109853 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.109807    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c" podUID="56a940cb-a4a0-4593-a5a7-04883f9cab24" containerName="main" containerID="cri-o://19daf8bc10a6559d7f2c84b7d613751f94694cd2d43d93c902f652d2a01cda59" gracePeriod=45
Nov 02 13:23:54.235302 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.235273    4464 generic.go:356] "Generic (PLEG): container finished" podID="56a940cb-a4a0-4593-a5a7-04883f9cab24" containerID="19daf8bc10a6559d7f2c84b7d613751f94694cd2d43d93c902f652d2a01cda59" exitCode=0
Nov 02 13:23:54.235419 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.235317    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c" event={"ID":"56a940cb-a4a0-4593-a5a7-04883f9cab24","Type":"ContainerDied","Data":"19daf8bc10a6559d7f2c84b7d613751f94694cd2d43d93c902f652d2a01cda59"}
Nov 02 13:23:54.362373 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.362313    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:23:54.416719 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.416695    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-tmp-dir\") pod \"56a940cb-a4a0-4593-a5a7-04883f9cab24\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") "
Nov 02 13:23:54.416869 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.416747    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-dshm\") pod \"56a940cb-a4a0-4593-a5a7-04883f9cab24\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") "
Nov 02 13:23:54.416869 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.416778    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a940cb-a4a0-4593-a5a7-04883f9cab24-kube-api-access-6tn47\" (UniqueName: \"kubernetes.io/projected/56a940cb-a4a0-4593-a5a7-04883f9cab24-kube-api-access-6tn47\") pod \"56a940cb-a4a0-4593-a5a7-04883f9cab24\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") "
Nov 02 13:23:54.416869 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.416846    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-home\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-home\") pod \"56a940cb-a4a0-4593-a5a7-04883f9cab24\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") "
Nov 02 13:23:54.417009 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.416895    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a940cb-a4a0-4593-a5a7-04883f9cab24-tls-certs\" (UniqueName: \"kubernetes.io/secret/56a940cb-a4a0-4593-a5a7-04883f9cab24-tls-certs\") pod \"56a940cb-a4a0-4593-a5a7-04883f9cab24\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") "
Nov 02 13:23:54.417009 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.416941    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-model-cache\") pod \"56a940cb-a4a0-4593-a5a7-04883f9cab24\" (UID: \"56a940cb-a4a0-4593-a5a7-04883f9cab24\") "
Nov 02 13:23:54.417132 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.417003    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-tmp-dir" pod "56a940cb-a4a0-4593-a5a7-04883f9cab24" (UID: "56a940cb-a4a0-4593-a5a7-04883f9cab24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:54.417210 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.417184    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-home" pod "56a940cb-a4a0-4593-a5a7-04883f9cab24" (UID: "56a940cb-a4a0-4593-a5a7-04883f9cab24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:54.417210 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.417198    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-model-cache" pod "56a940cb-a4a0-4593-a5a7-04883f9cab24" (UID: "56a940cb-a4a0-4593-a5a7-04883f9cab24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:54.417313 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.417256    4464 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-tmp-dir\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:23:54.418852 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.418821    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-dshm" pod "56a940cb-a4a0-4593-a5a7-04883f9cab24" (UID: "56a940cb-a4a0-4593-a5a7-04883f9cab24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:54.419003 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.418949    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a940cb-a4a0-4593-a5a7-04883f9cab24-tls-certs" pod "56a940cb-a4a0-4593-a5a7-04883f9cab24" (UID: "56a940cb-a4a0-4593-a5a7-04883f9cab24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:23:54.419003 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.418961    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a940cb-a4a0-4593-a5a7-04883f9cab24-kube-api-access-6tn47" pod "56a940cb-a4a0-4593-a5a7-04883f9cab24" (UID: "56a940cb-a4a0-4593-a5a7-04883f9cab24"). InnerVolumeSpecName "kube-api-access-6tn47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:23:54.518577 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.518546    4464 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-model-cache\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:23:54.518577 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.518574    4464 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-dshm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:23:54.518577 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.518584    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tn47\" (UniqueName: \"kubernetes.io/projected/56a940cb-a4a0-4593-a5a7-04883f9cab24-kube-api-access-6tn47\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:23:54.518819 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.518593    4464 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56a940cb-a4a0-4593-a5a7-04883f9cab24-home\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:23:54.518819 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:54.518602    4464 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56a940cb-a4a0-4593-a5a7-04883f9cab24-tls-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:23:55.242033 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:55.242005    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"
Nov 02 13:23:55.242033 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:55.242021    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c" event={"ID":"56a940cb-a4a0-4593-a5a7-04883f9cab24","Type":"ContainerDied","Data":"f8250ac791eda078fa6392dec526b8470e4a30e085175adddcc1266d6006bc3d"}
Nov 02 13:23:55.242547 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:55.242072    4464 scope.go:122] "RemoveContainer" containerID="19daf8bc10a6559d7f2c84b7d613751f94694cd2d43d93c902f652d2a01cda59"
Nov 02 13:23:55.408392 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:55.408358    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"]
Nov 02 13:23:55.459161 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:55.459132    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-128a019f-kserve-8758f7f8f-dqm9c"]
Nov 02 13:23:55.581594 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:55.581516    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a940cb-a4a0-4593-a5a7-04883f9cab24" path="/var/lib/kubelet/pods/56a940cb-a4a0-4593-a5a7-04883f9cab24/volumes"
Nov 02 13:23:55.822916 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:23:55.822880    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:23:58.367755 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:23:58.367705    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:24:07.406747 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:07.406713    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:24:08.860626 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.860593    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"]
Nov 02 13:24:08.861033 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.861013    4464 cpu_manager.go:414] "containerMap: removing container" podUID="7901efe8-9ffe-4931-8185-3112b79085a0" containerName="revoke-keys"
Nov 02 13:24:08.861033 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.861024    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7901efe8-9ffe-4931-8185-3112b79085a0" containerName="revoke-keys"
Nov 02 13:24:08.861122 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.861040    4464 cpu_manager.go:414] "containerMap: removing container" podUID="56a940cb-a4a0-4593-a5a7-04883f9cab24" containerName="main"
Nov 02 13:24:08.861122 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.861046    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a940cb-a4a0-4593-a5a7-04883f9cab24" containerName="main"
Nov 02 13:24:08.861122 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.861117    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="7901efe8-9ffe-4931-8185-3112b79085a0" containerName="revoke-keys"
Nov 02 13:24:08.861233 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.861128    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a940cb-a4a0-4593-a5a7-04883f9cab24" containerName="main"
Nov 02 13:24:08.864268 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.864252    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"
Nov 02 13:24:08.884739 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.884708    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bfng\" (UniqueName: \"kubernetes.io/projected/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-kube-api-access-5bfng\") pod \"maas-api-revoke-sub-dc4495d7-bhtr2\" (UID: \"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"
Nov 02 13:24:08.884862 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.884831    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc4495d7-bhtr2\" (UID: \"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"
Nov 02 13:24:08.884937 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.884918    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc4495d7-bhtr2\" (UID: \"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"
Nov 02 13:24:08.917535 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.917512    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:24:08.968242 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.968212    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:24:08.968242 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.968212    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:24:08.968430 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.968262    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:24:08.970447 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.970427    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"]
Nov 02 13:24:08.986265 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.986241    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc4495d7-bhtr2\" (UID: \"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"
Nov 02 13:24:08.986377 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.986294    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bfng\" (UniqueName: \"kubernetes.io/projected/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-kube-api-access-5bfng\") pod \"maas-api-revoke-sub-dc4495d7-bhtr2\" (UID: \"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"
Nov 02 13:24:08.986377 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.986350    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc4495d7-bhtr2\" (UID: \"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"
Nov 02 13:24:08.986965 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.986948    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc4495d7-bhtr2\" (UID: \"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"
Nov 02 13:24:08.988592 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:08.988570    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc4495d7-bhtr2\" (UID: \"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"
Nov 02 13:24:09.050277 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:09.050253    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bfng\" (UniqueName: \"kubernetes.io/projected/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-kube-api-access-5bfng\") pod \"maas-api-revoke-sub-dc4495d7-bhtr2\" (UID: \"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"
Nov 02 13:24:09.176959 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:09.176895    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"
Nov 02 13:24:09.548361 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:09.548332    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"]
Nov 02 13:24:09.548810 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:24:09.548788    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2c581d_6d4d_40a1_9348_12f0e3fb6b38.slice/crio-6315ccefb8c4f665c922df7c21d72233f8d246f1d99147ac577e7349db708ad6 WatchSource:0}: Error finding container 6315ccefb8c4f665c922df7c21d72233f8d246f1d99147ac577e7349db708ad6: Status 404 returned error can't find the container with id 6315ccefb8c4f665c922df7c21d72233f8d246f1d99147ac577e7349db708ad6
Nov 02 13:24:10.365746 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:10.365701    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2" event={"ID":"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38","Type":"ContainerStarted","Data":"0675c9ecac3ab0b85683c8b017f09d598fd74b586756f49dd2a3dcf29ef73271"}
Nov 02 13:24:10.366170 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:10.365765    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2" event={"ID":"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38","Type":"ContainerStarted","Data":"6315ccefb8c4f665c922df7c21d72233f8d246f1d99147ac577e7349db708ad6"}
Nov 02 13:24:10.484576 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:10.484524    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2" podStartSLOduration=2.4845097320000002 podStartE2EDuration="2.484509732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:24:10.483923332 +0000 UTC m=+5279.047631737" watchObservedRunningTime="2026-11-02 13:24:10.484509732 +0000 UTC m=+5279.048218137"
Nov 02 13:24:11.372441 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:11.372403    4464 generic.go:356] "Generic (PLEG): container finished" podID="8a2c581d-6d4d-40a1-9348-12f0e3fb6b38" containerID="0675c9ecac3ab0b85683c8b017f09d598fd74b586756f49dd2a3dcf29ef73271" exitCode=0
Nov 02 13:24:11.372886 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:11.372484    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2" event={"ID":"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38","Type":"ContainerDied","Data":"0675c9ecac3ab0b85683c8b017f09d598fd74b586756f49dd2a3dcf29ef73271"}
Nov 02 13:24:12.519692 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:12.519667    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"
Nov 02 13:24:12.633471 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:12.633440    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-kube-api-access-5bfng\" (UniqueName: \"kubernetes.io/projected/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-kube-api-access-5bfng\") pod \"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38\" (UID: \"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38\") "
Nov 02 13:24:12.633699 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:12.633531    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-maas-api-openshift-ca\") pod \"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38\" (UID: \"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38\") "
Nov 02 13:24:12.633699 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:12.633588    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-maas-api-xks-ca\") pod \"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38\" (UID: \"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38\") "
Nov 02 13:24:12.633938 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:12.633909    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-maas-api-openshift-ca" pod "8a2c581d-6d4d-40a1-9348-12f0e3fb6b38" (UID: "8a2c581d-6d4d-40a1-9348-12f0e3fb6b38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:24:12.635589 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:12.635560    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-kube-api-access-5bfng" pod "8a2c581d-6d4d-40a1-9348-12f0e3fb6b38" (UID: "8a2c581d-6d4d-40a1-9348-12f0e3fb6b38"). InnerVolumeSpecName "kube-api-access-5bfng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:24:12.635677 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:12.635595    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-maas-api-xks-ca" pod "8a2c581d-6d4d-40a1-9348-12f0e3fb6b38" (UID: "8a2c581d-6d4d-40a1-9348-12f0e3fb6b38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:24:12.734420 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:12.734379    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:24:12.734420 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:12.734413    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:24:12.734420 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:12.734426    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bfng\" (UniqueName: \"kubernetes.io/projected/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38-kube-api-access-5bfng\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:24:13.384600 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:13.384568    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2" event={"ID":"8a2c581d-6d4d-40a1-9348-12f0e3fb6b38","Type":"ContainerDied","Data":"6315ccefb8c4f665c922df7c21d72233f8d246f1d99147ac577e7349db708ad6"}
Nov 02 13:24:13.384785 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:13.384618    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6315ccefb8c4f665c922df7c21d72233f8d246f1d99147ac577e7349db708ad6"
Nov 02 13:24:13.384785 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:13.384585    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"
Nov 02 13:24:14.796308 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:14.796280    4464 scope.go:122] "RemoveContainer" containerID="e08307775246c1ad69f11e51e17a194fad04ef330eb681d8ba59525692b237ee"
Nov 02 13:24:14.805041 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:14.805023    4464 scope.go:122] "RemoveContainer" containerID="2c7a279d7240198af2e3df14058b9898aeaaf3fa93cb7195b63cb814133b7dfa"
Nov 02 13:24:21.692568 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:21.692534    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:24:25.117281 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.117245    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"]
Nov 02 13:24:25.117738 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.117723    4464 cpu_manager.go:414] "containerMap: removing container" podUID="8a2c581d-6d4d-40a1-9348-12f0e3fb6b38" containerName="revoke-keys"
Nov 02 13:24:25.117738 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.117736    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a2c581d-6d4d-40a1-9348-12f0e3fb6b38" containerName="revoke-keys"
Nov 02 13:24:25.117848 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.117819    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a2c581d-6d4d-40a1-9348-12f0e3fb6b38" containerName="revoke-keys"
Nov 02 13:24:25.122490 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.122471    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.177711 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.177677    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1e6e34-openshift-default-dockercfg-f7ttq\""
Nov 02 13:24:25.230201 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.230166    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"]
Nov 02 13:24:25.258925 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.258878    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-workload-socket\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.259160 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.258955    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-token\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.259160 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.259011    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8tqz\" (UniqueName: \"kubernetes.io/projected/704b4f8b-49a9-4b4f-b6bc-81d48164343c-kube-api-access-f8tqz\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.259160 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.259076    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-envoy\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.259339 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.259163    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-credential-socket\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.259339 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.259279    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-workload-certs\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.259339 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.259319    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-data\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.272163 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.272128    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istiod-ca-cert\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.272373 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.272185    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-podinfo\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.373808 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.373720    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8tqz\" (UniqueName: \"kubernetes.io/projected/704b4f8b-49a9-4b4f-b6bc-81d48164343c-kube-api-access-f8tqz\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.373808 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.373779    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-envoy\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.374084 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.373824    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-credential-socket\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.374084 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.373852    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-workload-certs\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.374084 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.373884    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-data\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.374084 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.373919    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istiod-ca-cert\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.374084 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.373951    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-podinfo\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.374084 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.374056    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-workload-socket\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.374435 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.374111    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-token\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.374435 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.374279    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-credential-socket\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.374586 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.374557    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-workload-certs\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.374586 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.374574    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-data\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.374814 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.374780    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istiod-ca-cert\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.374885 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.374811    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-workload-socket\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.376207 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.376188    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-envoy\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.376508 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.376491    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-podinfo\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.437674 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.437639    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-token\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.437838 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.437725    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8tqz\" (UniqueName: \"kubernetes.io/projected/704b4f8b-49a9-4b4f-b6bc-81d48164343c-kube-api-access-f8tqz\") pod \"e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") " pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:25.735467 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:25.735436    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:26.212085 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:26.212051    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"]
Nov 02 13:24:26.213743 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:24:26.213714    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704b4f8b_49a9_4b4f_b6bc_81d48164343c.slice/crio-b5d0ca03f8783b75d35aa8b9e333f69bee3ae2a5ca64d9f8c235dd2530477c6c WatchSource:0}: Error finding container b5d0ca03f8783b75d35aa8b9e333f69bee3ae2a5ca64d9f8c235dd2530477c6c: Status 404 returned error can't find the container with id b5d0ca03f8783b75d35aa8b9e333f69bee3ae2a5ca64d9f8c235dd2530477c6c
Nov 02 13:24:26.215818 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:26.215787    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:24:26.215934 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:26.215868    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:24:26.215934 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:26.215909    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:24:26.452122 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:26.452074    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z" event={"ID":"704b4f8b-49a9-4b4f-b6bc-81d48164343c","Type":"ContainerStarted","Data":"a46433bcd0d378e8ed6b48154b4cd1fdaa6830fa4ca2f712aba82856356ec319"}
Nov 02 13:24:26.452297 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:26.452149    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z" event={"ID":"704b4f8b-49a9-4b4f-b6bc-81d48164343c","Type":"ContainerStarted","Data":"b5d0ca03f8783b75d35aa8b9e333f69bee3ae2a5ca64d9f8c235dd2530477c6c"}
Nov 02 13:24:26.576816 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:26.576765    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z" podStartSLOduration=1.576747482 podStartE2EDuration="1.576747482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:24:26.574776076 +0000 UTC m=+5295.138484510" watchObservedRunningTime="2026-11-02 13:24:26.576747482 +0000 UTC m=+5295.140455890"
Nov 02 13:24:26.735893 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:26.735863    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:27.741361 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:27.741331    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:28.292166 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:24:28.292073    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:24:28.462417 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:28.462392    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:28.463183 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:28.463160    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:33.345995 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.342695    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l"]
Nov 02 13:24:33.349996 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.346855    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l"
Nov 02 13:24:33.370823 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.370779    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80087027-5e25-44db-a362-02a95a499e77-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l\" (UID: \"80087027-5e25-44db-a362-02a95a499e77\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l"
Nov 02 13:24:33.371079 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.371010    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8jst\" (UniqueName: \"kubernetes.io/projected/80087027-5e25-44db-a362-02a95a499e77-kube-api-access-q8jst\") pod \"payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l\" (UID: \"80087027-5e25-44db-a362-02a95a499e77\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l"
Nov 02 13:24:33.451563 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.451529    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1e6e34\""
Nov 02 13:24:33.451563 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.451530    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1e6e34-dockercfg-klp9q\""
Nov 02 13:24:33.453910 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.453890    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l"]
Nov 02 13:24:33.471790 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.471769    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8jst\" (UniqueName: \"kubernetes.io/projected/80087027-5e25-44db-a362-02a95a499e77-kube-api-access-q8jst\") pod \"payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l\" (UID: \"80087027-5e25-44db-a362-02a95a499e77\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l"
Nov 02 13:24:33.471898 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.471843    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80087027-5e25-44db-a362-02a95a499e77-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l\" (UID: \"80087027-5e25-44db-a362-02a95a499e77\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l"
Nov 02 13:24:33.472506 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.472485    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80087027-5e25-44db-a362-02a95a499e77-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l\" (UID: \"80087027-5e25-44db-a362-02a95a499e77\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l"
Nov 02 13:24:33.522766 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.522744    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff"]
Nov 02 13:24:33.526518 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.526500    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff"
Nov 02 13:24:33.534437 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.534418    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8jst\" (UniqueName: \"kubernetes.io/projected/80087027-5e25-44db-a362-02a95a499e77-kube-api-access-q8jst\") pod \"payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l\" (UID: \"80087027-5e25-44db-a362-02a95a499e77\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l"
Nov 02 13:24:33.572650 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.572628    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drg68\" (UniqueName: \"kubernetes.io/projected/e0cc3b4f-3b82-4e80-a4e8-9c191a604981-kube-api-access-drg68\") pod \"payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff\" (UID: \"e0cc3b4f-3b82-4e80-a4e8-9c191a604981\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff"
Nov 02 13:24:33.572765 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.572685    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0cc3b4f-3b82-4e80-a4e8-9c191a604981-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff\" (UID: \"e0cc3b4f-3b82-4e80-a4e8-9c191a604981\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff"
Nov 02 13:24:33.585633 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.585611    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff"]
Nov 02 13:24:33.673764 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.673691    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drg68\" (UniqueName: \"kubernetes.io/projected/e0cc3b4f-3b82-4e80-a4e8-9c191a604981-kube-api-access-drg68\") pod \"payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff\" (UID: \"e0cc3b4f-3b82-4e80-a4e8-9c191a604981\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff"
Nov 02 13:24:33.673907 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.673769    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0cc3b4f-3b82-4e80-a4e8-9c191a604981-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff\" (UID: \"e0cc3b4f-3b82-4e80-a4e8-9c191a604981\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff"
Nov 02 13:24:33.674460 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.674441    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0cc3b4f-3b82-4e80-a4e8-9c191a604981-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff\" (UID: \"e0cc3b4f-3b82-4e80-a4e8-9c191a604981\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff"
Nov 02 13:24:33.680541 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.680525    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l"
Nov 02 13:24:33.741508 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.736830    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drg68\" (UniqueName: \"kubernetes.io/projected/e0cc3b4f-3b82-4e80-a4e8-9c191a604981-kube-api-access-drg68\") pod \"payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff\" (UID: \"e0cc3b4f-3b82-4e80-a4e8-9c191a604981\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff"
Nov 02 13:24:33.839777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.839745    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff"
Nov 02 13:24:33.951395 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:33.951330    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l"]
Nov 02 13:24:34.051689 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:34.051651    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff"]
Nov 02 13:24:34.062991 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:34.062959    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l"]
Nov 02 13:24:34.064491 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:24:34.064467    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80087027_5e25_44db_a362_02a95a499e77.slice/crio-4d3ec1f037bc015783e67badb60668545c734919deed5e222a97c2b0d7bba574 WatchSource:0}: Error finding container 4d3ec1f037bc015783e67badb60668545c734919deed5e222a97c2b0d7bba574: Status 404 returned error can't find the container with id 4d3ec1f037bc015783e67badb60668545c734919deed5e222a97c2b0d7bba574
Nov 02 13:24:34.209515 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:34.209491    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff"]
Nov 02 13:24:34.210993 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:24:34.210958    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cc3b4f_3b82_4e80_a4e8_9c191a604981.slice/crio-270125d8b56cc6971eb8c407dbf37258bd61d6880f584ae1be861d3b5451fab0 WatchSource:0}: Error finding container 270125d8b56cc6971eb8c407dbf37258bd61d6880f584ae1be861d3b5451fab0: Status 404 returned error can't find the container with id 270125d8b56cc6971eb8c407dbf37258bd61d6880f584ae1be861d3b5451fab0
Nov 02 13:24:34.510147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:34.510106    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l" event={"ID":"80087027-5e25-44db-a362-02a95a499e77","Type":"ContainerStarted","Data":"0d06deb090160fb86721ea05bcde1d80b2d277269eb587482843c7f4cc31ae06"}
Nov 02 13:24:34.510642 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:34.510175    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l" event={"ID":"80087027-5e25-44db-a362-02a95a499e77","Type":"ContainerStarted","Data":"4d3ec1f037bc015783e67badb60668545c734919deed5e222a97c2b0d7bba574"}
Nov 02 13:24:34.510642 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:34.510207    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l"
Nov 02 13:24:34.512062 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:34.512032    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff" event={"ID":"e0cc3b4f-3b82-4e80-a4e8-9c191a604981","Type":"ContainerStarted","Data":"5f4d82e00d69e008020ee5cbabee2c4ce0a3c703a80235ecd532284c8ee3144b"}
Nov 02 13:24:34.512219 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:34.512080    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff" event={"ID":"e0cc3b4f-3b82-4e80-a4e8-9c191a604981","Type":"ContainerStarted","Data":"270125d8b56cc6971eb8c407dbf37258bd61d6880f584ae1be861d3b5451fab0"}
Nov 02 13:24:34.512219 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:34.512141    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff"
Nov 02 13:24:34.630955 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:34.630889    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l" podStartSLOduration=1.630871599 podStartE2EDuration="1.630871599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:24:34.627947093 +0000 UTC m=+5303.191655503" watchObservedRunningTime="2026-11-02 13:24:34.630871599 +0000 UTC m=+5303.194580009"
Nov 02 13:24:34.748133 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:34.748079    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff" podStartSLOduration=1.748064611 podStartE2EDuration="1.748064611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:24:34.746104407 +0000 UTC m=+5303.309812844" watchObservedRunningTime="2026-11-02 13:24:34.748064611 +0000 UTC m=+5303.311773020"
Nov 02 13:24:39.510391 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:39.510348    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l" podUID="80087027-5e25-44db-a362-02a95a499e77" containerName="payload-processing" containerID="cri-o://0d06deb090160fb86721ea05bcde1d80b2d277269eb587482843c7f4cc31ae06" gracePeriod=25
Nov 02 13:24:39.512896 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:39.512847    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff" podUID="e0cc3b4f-3b82-4e80-a4e8-9c191a604981" containerName="payload-pre-processing" containerID="cri-o://5f4d82e00d69e008020ee5cbabee2c4ce0a3c703a80235ecd532284c8ee3144b" gracePeriod=25
Nov 02 13:24:39.764582 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:39.764522    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff"
Nov 02 13:24:39.787815 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:39.787795    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l"
Nov 02 13:24:39.834283 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:39.834255    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80087027-5e25-44db-a362-02a95a499e77-kube-api-access-q8jst\" (UniqueName: \"kubernetes.io/projected/80087027-5e25-44db-a362-02a95a499e77-kube-api-access-q8jst\") pod \"80087027-5e25-44db-a362-02a95a499e77\" (UID: \"80087027-5e25-44db-a362-02a95a499e77\") "
Nov 02 13:24:39.834435 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:39.834389    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0cc3b4f-3b82-4e80-a4e8-9c191a604981-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0cc3b4f-3b82-4e80-a4e8-9c191a604981-plugins-config-volume\") pod \"e0cc3b4f-3b82-4e80-a4e8-9c191a604981\" (UID: \"e0cc3b4f-3b82-4e80-a4e8-9c191a604981\") "
Nov 02 13:24:39.834435 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:39.834413    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80087027-5e25-44db-a362-02a95a499e77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80087027-5e25-44db-a362-02a95a499e77-plugins-config-volume\") pod \"80087027-5e25-44db-a362-02a95a499e77\" (UID: \"80087027-5e25-44db-a362-02a95a499e77\") "
Nov 02 13:24:39.834435 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:39.834430    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0cc3b4f-3b82-4e80-a4e8-9c191a604981-kube-api-access-drg68\" (UniqueName: \"kubernetes.io/projected/e0cc3b4f-3b82-4e80-a4e8-9c191a604981-kube-api-access-drg68\") pod \"e0cc3b4f-3b82-4e80-a4e8-9c191a604981\" (UID: \"e0cc3b4f-3b82-4e80-a4e8-9c191a604981\") "
Nov 02 13:24:39.834812 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:39.834789    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0cc3b4f-3b82-4e80-a4e8-9c191a604981-plugins-config-volume" pod "e0cc3b4f-3b82-4e80-a4e8-9c191a604981" (UID: "e0cc3b4f-3b82-4e80-a4e8-9c191a604981"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:24:39.834871 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:39.834801    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80087027-5e25-44db-a362-02a95a499e77-plugins-config-volume" pod "80087027-5e25-44db-a362-02a95a499e77" (UID: "80087027-5e25-44db-a362-02a95a499e77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:24:39.836354 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:39.836330    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80087027-5e25-44db-a362-02a95a499e77-kube-api-access-q8jst" pod "80087027-5e25-44db-a362-02a95a499e77" (UID: "80087027-5e25-44db-a362-02a95a499e77"). InnerVolumeSpecName "kube-api-access-q8jst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:24:39.836436 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:39.836402    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0cc3b4f-3b82-4e80-a4e8-9c191a604981-kube-api-access-drg68" pod "e0cc3b4f-3b82-4e80-a4e8-9c191a604981" (UID: "e0cc3b4f-3b82-4e80-a4e8-9c191a604981"). InnerVolumeSpecName "kube-api-access-drg68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:24:39.936114 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:39.936077    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80087027-5e25-44db-a362-02a95a499e77-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:24:39.936114 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:39.936107    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drg68\" (UniqueName: \"kubernetes.io/projected/e0cc3b4f-3b82-4e80-a4e8-9c191a604981-kube-api-access-drg68\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:24:39.936114 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:39.936116    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8jst\" (UniqueName: \"kubernetes.io/projected/80087027-5e25-44db-a362-02a95a499e77-kube-api-access-q8jst\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:24:39.936388 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:39.936125    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0cc3b4f-3b82-4e80-a4e8-9c191a604981-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:24:40.544941 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.544905    4464 generic.go:356] "Generic (PLEG): container finished" podID="e0cc3b4f-3b82-4e80-a4e8-9c191a604981" containerID="5f4d82e00d69e008020ee5cbabee2c4ce0a3c703a80235ecd532284c8ee3144b" exitCode=0
Nov 02 13:24:40.545370 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.545007    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff"
Nov 02 13:24:40.545370 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.545014    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff" event={"ID":"e0cc3b4f-3b82-4e80-a4e8-9c191a604981","Type":"ContainerDied","Data":"5f4d82e00d69e008020ee5cbabee2c4ce0a3c703a80235ecd532284c8ee3144b"}
Nov 02 13:24:40.545370 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.545191    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff" event={"ID":"e0cc3b4f-3b82-4e80-a4e8-9c191a604981","Type":"ContainerDied","Data":"270125d8b56cc6971eb8c407dbf37258bd61d6880f584ae1be861d3b5451fab0"}
Nov 02 13:24:40.545370 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.545216    4464 scope.go:122] "RemoveContainer" containerID="5f4d82e00d69e008020ee5cbabee2c4ce0a3c703a80235ecd532284c8ee3144b"
Nov 02 13:24:40.546654 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.546544    4464 generic.go:356] "Generic (PLEG): container finished" podID="80087027-5e25-44db-a362-02a95a499e77" containerID="0d06deb090160fb86721ea05bcde1d80b2d277269eb587482843c7f4cc31ae06" exitCode=0
Nov 02 13:24:40.546654 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.546589    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l" event={"ID":"80087027-5e25-44db-a362-02a95a499e77","Type":"ContainerDied","Data":"0d06deb090160fb86721ea05bcde1d80b2d277269eb587482843c7f4cc31ae06"}
Nov 02 13:24:40.546654 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.546602    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l"
Nov 02 13:24:40.546865 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.546656    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l" event={"ID":"80087027-5e25-44db-a362-02a95a499e77","Type":"ContainerDied","Data":"4d3ec1f037bc015783e67badb60668545c734919deed5e222a97c2b0d7bba574"}
Nov 02 13:24:40.555431 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.555414    4464 scope.go:122] "RemoveContainer" containerID="5f4d82e00d69e008020ee5cbabee2c4ce0a3c703a80235ecd532284c8ee3144b"
Nov 02 13:24:40.555692 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:24:40.555670    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4d82e00d69e008020ee5cbabee2c4ce0a3c703a80235ecd532284c8ee3144b\": container with ID starting with 5f4d82e00d69e008020ee5cbabee2c4ce0a3c703a80235ecd532284c8ee3144b not found: ID does not exist" containerID="5f4d82e00d69e008020ee5cbabee2c4ce0a3c703a80235ecd532284c8ee3144b"
Nov 02 13:24:40.555804 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.555697    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4d82e00d69e008020ee5cbabee2c4ce0a3c703a80235ecd532284c8ee3144b"} err="failed to get container status \"5f4d82e00d69e008020ee5cbabee2c4ce0a3c703a80235ecd532284c8ee3144b\": rpc error: code = NotFound desc = could not find container \"5f4d82e00d69e008020ee5cbabee2c4ce0a3c703a80235ecd532284c8ee3144b\": container with ID starting with 5f4d82e00d69e008020ee5cbabee2c4ce0a3c703a80235ecd532284c8ee3144b not found: ID does not exist"
Nov 02 13:24:40.555804 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.555713    4464 scope.go:122] "RemoveContainer" containerID="0d06deb090160fb86721ea05bcde1d80b2d277269eb587482843c7f4cc31ae06"
Nov 02 13:24:40.563834 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.563817    4464 scope.go:122] "RemoveContainer" containerID="0d06deb090160fb86721ea05bcde1d80b2d277269eb587482843c7f4cc31ae06"
Nov 02 13:24:40.564156 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:24:40.564135    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d06deb090160fb86721ea05bcde1d80b2d277269eb587482843c7f4cc31ae06\": container with ID starting with 0d06deb090160fb86721ea05bcde1d80b2d277269eb587482843c7f4cc31ae06 not found: ID does not exist" containerID="0d06deb090160fb86721ea05bcde1d80b2d277269eb587482843c7f4cc31ae06"
Nov 02 13:24:40.564241 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.564162    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d06deb090160fb86721ea05bcde1d80b2d277269eb587482843c7f4cc31ae06"} err="failed to get container status \"0d06deb090160fb86721ea05bcde1d80b2d277269eb587482843c7f4cc31ae06\": rpc error: code = NotFound desc = could not find container \"0d06deb090160fb86721ea05bcde1d80b2d277269eb587482843c7f4cc31ae06\": container with ID starting with 0d06deb090160fb86721ea05bcde1d80b2d277269eb587482843c7f4cc31ae06 not found: ID does not exist"
Nov 02 13:24:40.712167 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.712137    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff"]
Nov 02 13:24:40.762991 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.762957    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-d9bdd6775-cxnff"]
Nov 02 13:24:40.830293 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.830219    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l"]
Nov 02 13:24:40.835066 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:40.835037    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-5c87746cbf-jnk6l"]
Nov 02 13:24:41.586242 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:41.586209    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80087027-5e25-44db-a362-02a95a499e77" path="/var/lib/kubelet/pods/80087027-5e25-44db-a362-02a95a499e77/volumes"
Nov 02 13:24:41.586887 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:41.586802    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0cc3b4f-3b82-4e80-a4e8-9c191a604981" path="/var/lib/kubelet/pods/e0cc3b4f-3b82-4e80-a4e8-9c191a604981/volumes"
Nov 02 13:24:46.935264 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:46.935226    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"]
Nov 02 13:24:46.935692 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:46.935679    4464 cpu_manager.go:414] "containerMap: removing container" podUID="80087027-5e25-44db-a362-02a95a499e77" containerName="payload-processing"
Nov 02 13:24:46.935740 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:46.935693    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80087027-5e25-44db-a362-02a95a499e77" containerName="payload-processing"
Nov 02 13:24:46.935740 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:46.935710    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e0cc3b4f-3b82-4e80-a4e8-9c191a604981" containerName="payload-pre-processing"
Nov 02 13:24:46.935740 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:46.935715    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0cc3b4f-3b82-4e80-a4e8-9c191a604981" containerName="payload-pre-processing"
Nov 02 13:24:46.935844 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:46.935793    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="80087027-5e25-44db-a362-02a95a499e77" containerName="payload-processing"
Nov 02 13:24:46.935844 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:46.935803    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0cc3b4f-3b82-4e80-a4e8-9c191a604981" containerName="payload-pre-processing"
Nov 02 13:24:46.940460 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:46.940441    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"
Nov 02 13:24:47.031909 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.031877    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhckf\" (UniqueName: \"kubernetes.io/projected/77fca781-cb8d-4524-8d83-16ccd4ced2f1-kube-api-access-hhckf\") pod \"payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w\" (UID: \"77fca781-cb8d-4524-8d83-16ccd4ced2f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"
Nov 02 13:24:47.032109 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.031917    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77fca781-cb8d-4524-8d83-16ccd4ced2f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w\" (UID: \"77fca781-cb8d-4524-8d83-16ccd4ced2f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"
Nov 02 13:24:47.044959 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.044935    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1e6e34-dockercfg-p9rx7\""
Nov 02 13:24:47.045059 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.044990    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1e6e34\""
Nov 02 13:24:47.047401 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.047378    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"]
Nov 02 13:24:47.133322 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.133289    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhckf\" (UniqueName: \"kubernetes.io/projected/77fca781-cb8d-4524-8d83-16ccd4ced2f1-kube-api-access-hhckf\") pod \"payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w\" (UID: \"77fca781-cb8d-4524-8d83-16ccd4ced2f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"
Nov 02 13:24:47.133511 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.133329    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77fca781-cb8d-4524-8d83-16ccd4ced2f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w\" (UID: \"77fca781-cb8d-4524-8d83-16ccd4ced2f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"
Nov 02 13:24:47.134022 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.134005    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77fca781-cb8d-4524-8d83-16ccd4ced2f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w\" (UID: \"77fca781-cb8d-4524-8d83-16ccd4ced2f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"
Nov 02 13:24:47.195462 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.195387    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhckf\" (UniqueName: \"kubernetes.io/projected/77fca781-cb8d-4524-8d83-16ccd4ced2f1-kube-api-access-hhckf\") pod \"payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w\" (UID: \"77fca781-cb8d-4524-8d83-16ccd4ced2f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"
Nov 02 13:24:47.253365 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.253332    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"
Nov 02 13:24:47.625411 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.625383    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"]
Nov 02 13:24:47.627495 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:24:47.627470    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77fca781_cb8d_4524_8d83_16ccd4ced2f1.slice/crio-f0d429b9d46cc2896cfc9229371b373b338875f388c85530692dfb72e1f3cd54 WatchSource:0}: Error finding container f0d429b9d46cc2896cfc9229371b373b338875f388c85530692dfb72e1f3cd54: Status 404 returned error can't find the container with id f0d429b9d46cc2896cfc9229371b373b338875f388c85530692dfb72e1f3cd54
Nov 02 13:24:47.629870 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.629848    4464 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:24:47.672301 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.672267    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:24:47.695670 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.695640    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"]
Nov 02 13:24:47.701170 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.701147    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"
Nov 02 13:24:47.757060 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.757029    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"]
Nov 02 13:24:47.850854 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.850813    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704f2bb8-c222-405d-985e-b471e50146d3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds\" (UID: \"704f2bb8-c222-405d-985e-b471e50146d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"
Nov 02 13:24:47.851064 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.851042    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2wz9\" (UniqueName: \"kubernetes.io/projected/704f2bb8-c222-405d-985e-b471e50146d3-kube-api-access-c2wz9\") pod \"payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds\" (UID: \"704f2bb8-c222-405d-985e-b471e50146d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"
Nov 02 13:24:47.952619 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.952524    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2wz9\" (UniqueName: \"kubernetes.io/projected/704f2bb8-c222-405d-985e-b471e50146d3-kube-api-access-c2wz9\") pod \"payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds\" (UID: \"704f2bb8-c222-405d-985e-b471e50146d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"
Nov 02 13:24:47.952619 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.952577    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704f2bb8-c222-405d-985e-b471e50146d3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds\" (UID: \"704f2bb8-c222-405d-985e-b471e50146d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"
Nov 02 13:24:47.953315 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:47.953295    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704f2bb8-c222-405d-985e-b471e50146d3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds\" (UID: \"704f2bb8-c222-405d-985e-b471e50146d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"
Nov 02 13:24:48.018272 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:48.018238    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2wz9\" (UniqueName: \"kubernetes.io/projected/704f2bb8-c222-405d-985e-b471e50146d3-kube-api-access-c2wz9\") pod \"payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds\" (UID: \"704f2bb8-c222-405d-985e-b471e50146d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"
Nov 02 13:24:48.317280 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:48.317246    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"
Nov 02 13:24:48.601029 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:48.600924    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w" event={"ID":"77fca781-cb8d-4524-8d83-16ccd4ced2f1","Type":"ContainerStarted","Data":"f0d429b9d46cc2896cfc9229371b373b338875f388c85530692dfb72e1f3cd54"}
Nov 02 13:24:48.692864 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:48.692837    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"]
Nov 02 13:24:48.694511 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:24:48.694481    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704f2bb8_c222_405d_985e_b471e50146d3.slice/crio-491bd4517f0323146bf0a0eb26c0b86747fc610ccdb4f778ee88e871044ee61e WatchSource:0}: Error finding container 491bd4517f0323146bf0a0eb26c0b86747fc610ccdb4f778ee88e871044ee61e: Status 404 returned error can't find the container with id 491bd4517f0323146bf0a0eb26c0b86747fc610ccdb4f778ee88e871044ee61e
Nov 02 13:24:49.609507 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:49.609464    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds" event={"ID":"704f2bb8-c222-405d-985e-b471e50146d3","Type":"ContainerStarted","Data":"491bd4517f0323146bf0a0eb26c0b86747fc610ccdb4f778ee88e871044ee61e"}
Nov 02 13:24:50.617056 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:50.617022    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w" event={"ID":"77fca781-cb8d-4524-8d83-16ccd4ced2f1","Type":"ContainerStarted","Data":"e1ba458984371348e6be9dcdc452118a7cd87098281b07927694fa828fbbccd4"}
Nov 02 13:24:50.617534 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:50.617144    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"
Nov 02 13:24:50.746835 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:50.746778    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w" podStartSLOduration=1.873923075 podStartE2EDuration="4.746764314s" totalImagesPullingTime="2.872841239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:24:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:24:50.744933919 +0000 UTC m=+5319.308642343" watchObservedRunningTime="2026-11-02 13:24:50.746764314 +0000 UTC m=+5319.310472722"
Nov 02 13:24:51.623502 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:51.623447    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds" event={"ID":"704f2bb8-c222-405d-985e-b471e50146d3","Type":"ContainerStarted","Data":"57640783b70350c5de412b9fac5a54f2de8b853c5b87be61ef8b134658318d1b"}
Nov 02 13:24:51.624002 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:51.623685    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"
Nov 02 13:24:51.745399 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:51.745345    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds" podStartSLOduration=2.690671091 podStartE2EDuration="4.745331837s" totalImagesPullingTime="2.054660746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:24:51.744693039 +0000 UTC m=+5320.308401453" watchObservedRunningTime="2026-11-02 13:24:51.745331837 +0000 UTC m=+5320.309040247"
Nov 02 13:24:57.656895 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:57.656863    4464 generic.go:356] "Generic (PLEG): container finished" podID="704b4f8b-49a9-4b4f-b6bc-81d48164343c" containerID="a46433bcd0d378e8ed6b48154b4cd1fdaa6830fa4ca2f712aba82856356ec319" exitCode=0
Nov 02 13:24:57.657398 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:57.656906    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z" event={"ID":"704b4f8b-49a9-4b4f-b6bc-81d48164343c","Type":"ContainerDied","Data":"a46433bcd0d378e8ed6b48154b4cd1fdaa6830fa4ca2f712aba82856356ec319"}
Nov 02 13:24:57.657398 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:57.657356    4464 scope.go:122] "RemoveContainer" containerID="a46433bcd0d378e8ed6b48154b4cd1fdaa6830fa4ca2f712aba82856356ec319"
Nov 02 13:24:57.658543 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:57.658512    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:24:57.658658 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:57.658619    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:24:57.658734 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:57.658669    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:24:58.292471 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:24:58.292416    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:24:58.462896 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:58.462752    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:58.669073 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:58.668777    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z" event={"ID":"704b4f8b-49a9-4b4f-b6bc-81d48164343c","Type":"ContainerStarted","Data":"c388057c7e09656da966e7dc54a7f4d8be47166f44183e7c71299a6db92af5d8"}
Nov 02 13:24:58.736577 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:58.736545    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:24:58.737607 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:58.737584    4464 patch_prober.go:29] interesting pod/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Nov 02 13:24:58.737727 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:58.737630    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z" podUID="704b4f8b-49a9-4b4f-b6bc-81d48164343c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Nov 02 13:24:59.736064 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:59.736034    4464 patch_prober.go:29] interesting pod/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Nov 02 13:24:59.736437 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:24:59.736082    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z" podUID="704b4f8b-49a9-4b4f-b6bc-81d48164343c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Nov 02 13:25:00.735783 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:00.735750    4464 patch_prober.go:29] interesting pod/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Nov 02 13:25:00.735960 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:00.735799    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z" podUID="704b4f8b-49a9-4b4f-b6bc-81d48164343c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Nov 02 13:25:01.626134 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:01.626104    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"
Nov 02 13:25:01.736053 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:01.736023    4464 patch_prober.go:29] interesting pod/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Nov 02 13:25:01.736226 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:01.736072    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z" podUID="704b4f8b-49a9-4b4f-b6bc-81d48164343c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Nov 02 13:25:02.631402 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:02.631373    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"
Nov 02 13:25:02.736121 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:02.736090    4464 patch_prober.go:29] interesting pod/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Nov 02 13:25:02.736294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:02.736143    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z" podUID="704b4f8b-49a9-4b4f-b6bc-81d48164343c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Nov 02 13:25:03.736202 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:03.736167    4464 patch_prober.go:29] interesting pod/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body=
Nov 02 13:25:03.736598 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:03.736232    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z" podUID="704b4f8b-49a9-4b4f-b6bc-81d48164343c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused"
Nov 02 13:25:04.740117 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:04.740091    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:25:04.740549 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:04.740466    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:25:04.741237 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:04.741221    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:25:14.333751 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:14.333701    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb"]
Nov 02 13:25:14.345995 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:14.344064    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb"
Nov 02 13:25:14.395430 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:14.395395    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb"]
Nov 02 13:25:14.442782 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:14.442751    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-193883-dockercfg-xdh52\""
Nov 02 13:25:14.442782 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:14.442751    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-193883\""
Nov 02 13:25:14.473313 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:14.473280    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86749a36-7d06-4af8-9e67-4f6ec1147ea5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb\" (UID: \"86749a36-7d06-4af8-9e67-4f6ec1147ea5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb"
Nov 02 13:25:14.473462 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:14.473356    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57fsw\" (UniqueName: \"kubernetes.io/projected/86749a36-7d06-4af8-9e67-4f6ec1147ea5-kube-api-access-57fsw\") pod \"payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb\" (UID: \"86749a36-7d06-4af8-9e67-4f6ec1147ea5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb"
Nov 02 13:25:14.574825 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:14.574782    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57fsw\" (UniqueName: \"kubernetes.io/projected/86749a36-7d06-4af8-9e67-4f6ec1147ea5-kube-api-access-57fsw\") pod \"payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb\" (UID: \"86749a36-7d06-4af8-9e67-4f6ec1147ea5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb"
Nov 02 13:25:14.575076 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:14.574957    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86749a36-7d06-4af8-9e67-4f6ec1147ea5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb\" (UID: \"86749a36-7d06-4af8-9e67-4f6ec1147ea5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb"
Nov 02 13:25:14.575663 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:14.575642    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86749a36-7d06-4af8-9e67-4f6ec1147ea5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb\" (UID: \"86749a36-7d06-4af8-9e67-4f6ec1147ea5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb"
Nov 02 13:25:14.638232 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:14.638159    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57fsw\" (UniqueName: \"kubernetes.io/projected/86749a36-7d06-4af8-9e67-4f6ec1147ea5-kube-api-access-57fsw\") pod \"payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb\" (UID: \"86749a36-7d06-4af8-9e67-4f6ec1147ea5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb"
Nov 02 13:25:14.677900 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:14.677861    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb"
Nov 02 13:25:14.974465 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:14.974434    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb"]
Nov 02 13:25:15.087028 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:15.087001    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb"]
Nov 02 13:25:15.089152 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:25:15.089126    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86749a36_7d06_4af8_9e67_4f6ec1147ea5.slice/crio-c9598d908c6a618daf3f53d27060496b072378e53ed5ae9a049f7a9ea567b61e WatchSource:0}: Error finding container c9598d908c6a618daf3f53d27060496b072378e53ed5ae9a049f7a9ea567b61e: Status 404 returned error can't find the container with id c9598d908c6a618daf3f53d27060496b072378e53ed5ae9a049f7a9ea567b61e
Nov 02 13:25:15.772247 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:15.772202    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb" event={"ID":"86749a36-7d06-4af8-9e67-4f6ec1147ea5","Type":"ContainerStarted","Data":"a9e43d511da7506d2ea6160421aefaca894b9b22be137139a68660ffdb7257aa"}
Nov 02 13:25:15.772640 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:15.772355    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb"
Nov 02 13:25:15.772640 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:15.772402    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb" event={"ID":"86749a36-7d06-4af8-9e67-4f6ec1147ea5","Type":"ContainerStarted","Data":"c9598d908c6a618daf3f53d27060496b072378e53ed5ae9a049f7a9ea567b61e"}
Nov 02 13:25:15.893790 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:15.893726    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb" podStartSLOduration=1.893708178 podStartE2EDuration="1.893708178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:25:15.891039612 +0000 UTC m=+5344.454748026" watchObservedRunningTime="2026-11-02 13:25:15.893708178 +0000 UTC m=+5344.457416588"
Nov 02 13:25:20.772876 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:20.772813    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb" podUID="86749a36-7d06-4af8-9e67-4f6ec1147ea5" containerName="payload-processing" containerID="cri-o://a9e43d511da7506d2ea6160421aefaca894b9b22be137139a68660ffdb7257aa" gracePeriod=25
Nov 02 13:25:21.024118 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:21.024054    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb"
Nov 02 13:25:21.144536 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:21.144499    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86749a36-7d06-4af8-9e67-4f6ec1147ea5-kube-api-access-57fsw\" (UniqueName: \"kubernetes.io/projected/86749a36-7d06-4af8-9e67-4f6ec1147ea5-kube-api-access-57fsw\") pod \"86749a36-7d06-4af8-9e67-4f6ec1147ea5\" (UID: \"86749a36-7d06-4af8-9e67-4f6ec1147ea5\") "
Nov 02 13:25:21.144936 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:21.144798    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86749a36-7d06-4af8-9e67-4f6ec1147ea5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86749a36-7d06-4af8-9e67-4f6ec1147ea5-plugins-config-volume\") pod \"86749a36-7d06-4af8-9e67-4f6ec1147ea5\" (UID: \"86749a36-7d06-4af8-9e67-4f6ec1147ea5\") "
Nov 02 13:25:21.145179 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:21.145122    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86749a36-7d06-4af8-9e67-4f6ec1147ea5-plugins-config-volume" pod "86749a36-7d06-4af8-9e67-4f6ec1147ea5" (UID: "86749a36-7d06-4af8-9e67-4f6ec1147ea5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:25:21.145634 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:21.145607    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86749a36-7d06-4af8-9e67-4f6ec1147ea5-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:25:21.146892 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:21.146860    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86749a36-7d06-4af8-9e67-4f6ec1147ea5-kube-api-access-57fsw" pod "86749a36-7d06-4af8-9e67-4f6ec1147ea5" (UID: "86749a36-7d06-4af8-9e67-4f6ec1147ea5"). InnerVolumeSpecName "kube-api-access-57fsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:25:21.246806 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:21.246770    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57fsw\" (UniqueName: \"kubernetes.io/projected/86749a36-7d06-4af8-9e67-4f6ec1147ea5-kube-api-access-57fsw\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:25:21.813715 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:21.813675    4464 generic.go:356] "Generic (PLEG): container finished" podID="86749a36-7d06-4af8-9e67-4f6ec1147ea5" containerID="a9e43d511da7506d2ea6160421aefaca894b9b22be137139a68660ffdb7257aa" exitCode=0
Nov 02 13:25:21.814196 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:21.813728    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb"
Nov 02 13:25:21.814196 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:21.813756    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb" event={"ID":"86749a36-7d06-4af8-9e67-4f6ec1147ea5","Type":"ContainerDied","Data":"a9e43d511da7506d2ea6160421aefaca894b9b22be137139a68660ffdb7257aa"}
Nov 02 13:25:21.814196 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:21.813810    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb" event={"ID":"86749a36-7d06-4af8-9e67-4f6ec1147ea5","Type":"ContainerDied","Data":"c9598d908c6a618daf3f53d27060496b072378e53ed5ae9a049f7a9ea567b61e"}
Nov 02 13:25:21.814196 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:21.813833    4464 scope.go:122] "RemoveContainer" containerID="a9e43d511da7506d2ea6160421aefaca894b9b22be137139a68660ffdb7257aa"
Nov 02 13:25:21.823075 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:21.823053    4464 scope.go:122] "RemoveContainer" containerID="a9e43d511da7506d2ea6160421aefaca894b9b22be137139a68660ffdb7257aa"
Nov 02 13:25:21.823326 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:25:21.823307    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9e43d511da7506d2ea6160421aefaca894b9b22be137139a68660ffdb7257aa\": container with ID starting with a9e43d511da7506d2ea6160421aefaca894b9b22be137139a68660ffdb7257aa not found: ID does not exist" containerID="a9e43d511da7506d2ea6160421aefaca894b9b22be137139a68660ffdb7257aa"
Nov 02 13:25:21.823382 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:21.823337    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e43d511da7506d2ea6160421aefaca894b9b22be137139a68660ffdb7257aa"} err="failed to get container status \"a9e43d511da7506d2ea6160421aefaca894b9b22be137139a68660ffdb7257aa\": rpc error: code = NotFound desc = could not find container \"a9e43d511da7506d2ea6160421aefaca894b9b22be137139a68660ffdb7257aa\": container with ID starting with a9e43d511da7506d2ea6160421aefaca894b9b22be137139a68660ffdb7257aa not found: ID does not exist"
Nov 02 13:25:21.980929 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:21.980895    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb"]
Nov 02 13:25:22.031693 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:22.031659    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-193883-5997b8d56c-nkmrb"]
Nov 02 13:25:23.581110 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:23.581076    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86749a36-7d06-4af8-9e67-4f6ec1147ea5" path="/var/lib/kubelet/pods/86749a36-7d06-4af8-9e67-4f6ec1147ea5/volumes"
Nov 02 13:25:28.292907 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:25:28.292820    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:25:30.240521 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:30.240487    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"]
Nov 02 13:25:30.241041 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:30.241024    4464 cpu_manager.go:414] "containerMap: removing container" podUID="86749a36-7d06-4af8-9e67-4f6ec1147ea5" containerName="payload-processing"
Nov 02 13:25:30.241118 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:30.241041    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86749a36-7d06-4af8-9e67-4f6ec1147ea5" containerName="payload-processing"
Nov 02 13:25:30.241160 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:30.241139    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="86749a36-7d06-4af8-9e67-4f6ec1147ea5" containerName="payload-processing"
Nov 02 13:25:30.245615 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:30.245599    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"
Nov 02 13:25:30.299903 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:30.299869    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-193883\""
Nov 02 13:25:30.339098 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:30.339065    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c6f60e-aecf-4f24-9d65-fe465b6e1734-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq\" (UID: \"23c6f60e-aecf-4f24-9d65-fe465b6e1734\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"
Nov 02 13:25:30.339270 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:30.339122    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq4ck\" (UniqueName: \"kubernetes.io/projected/23c6f60e-aecf-4f24-9d65-fe465b6e1734-kube-api-access-hq4ck\") pod \"payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq\" (UID: \"23c6f60e-aecf-4f24-9d65-fe465b6e1734\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"
Nov 02 13:25:30.350545 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:30.350510    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-193883-dockercfg-jhb24\""
Nov 02 13:25:30.352910 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:30.352883    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"]
Nov 02 13:25:30.440533 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:30.440495    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq4ck\" (UniqueName: \"kubernetes.io/projected/23c6f60e-aecf-4f24-9d65-fe465b6e1734-kube-api-access-hq4ck\") pod \"payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq\" (UID: \"23c6f60e-aecf-4f24-9d65-fe465b6e1734\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"
Nov 02 13:25:30.440704 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:30.440670    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c6f60e-aecf-4f24-9d65-fe465b6e1734-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq\" (UID: \"23c6f60e-aecf-4f24-9d65-fe465b6e1734\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"
Nov 02 13:25:30.441570 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:30.441546    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c6f60e-aecf-4f24-9d65-fe465b6e1734-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq\" (UID: \"23c6f60e-aecf-4f24-9d65-fe465b6e1734\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"
Nov 02 13:25:30.503202 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:30.503132    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq4ck\" (UniqueName: \"kubernetes.io/projected/23c6f60e-aecf-4f24-9d65-fe465b6e1734-kube-api-access-hq4ck\") pod \"payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq\" (UID: \"23c6f60e-aecf-4f24-9d65-fe465b6e1734\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"
Nov 02 13:25:30.559187 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:30.559158    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"
Nov 02 13:25:30.935049 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:30.935025    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"]
Nov 02 13:25:30.936845 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:25:30.936815    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c6f60e_aecf_4f24_9d65_fe465b6e1734.slice/crio-f9383e03a8074df33d89431447c90c19b23b8a43839f7f933fd24f18c738f708 WatchSource:0}: Error finding container f9383e03a8074df33d89431447c90c19b23b8a43839f7f933fd24f18c738f708: Status 404 returned error can't find the container with id f9383e03a8074df33d89431447c90c19b23b8a43839f7f933fd24f18c738f708
Nov 02 13:25:31.026597 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.026562    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"]
Nov 02 13:25:31.030909 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.030888    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"
Nov 02 13:25:31.045772 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.045744    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/517c3f5e-ced3-4482-a1a8-54d37cd2d75f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2\" (UID: \"517c3f5e-ced3-4482-a1a8-54d37cd2d75f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"
Nov 02 13:25:31.045907 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.045782    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn7zf\" (UniqueName: \"kubernetes.io/projected/517c3f5e-ced3-4482-a1a8-54d37cd2d75f-kube-api-access-jn7zf\") pod \"payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2\" (UID: \"517c3f5e-ced3-4482-a1a8-54d37cd2d75f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"
Nov 02 13:25:31.092552 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.092517    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"]
Nov 02 13:25:31.122944 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.122912    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:25:31.146906 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.146863    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/517c3f5e-ced3-4482-a1a8-54d37cd2d75f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2\" (UID: \"517c3f5e-ced3-4482-a1a8-54d37cd2d75f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"
Nov 02 13:25:31.147121 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.146916    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn7zf\" (UniqueName: \"kubernetes.io/projected/517c3f5e-ced3-4482-a1a8-54d37cd2d75f-kube-api-access-jn7zf\") pod \"payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2\" (UID: \"517c3f5e-ced3-4482-a1a8-54d37cd2d75f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"
Nov 02 13:25:31.147626 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.147595    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/517c3f5e-ced3-4482-a1a8-54d37cd2d75f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2\" (UID: \"517c3f5e-ced3-4482-a1a8-54d37cd2d75f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"
Nov 02 13:25:31.209554 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.209496    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn7zf\" (UniqueName: \"kubernetes.io/projected/517c3f5e-ced3-4482-a1a8-54d37cd2d75f-kube-api-access-jn7zf\") pod \"payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2\" (UID: \"517c3f5e-ced3-4482-a1a8-54d37cd2d75f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"
Nov 02 13:25:31.343458 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.343424    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"
Nov 02 13:25:31.716879 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.716856    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"]
Nov 02 13:25:31.717844 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:25:31.717806    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517c3f5e_ced3_4482_a1a8_54d37cd2d75f.slice/crio-3f31b6cba0833b20de60259cce82469ea204261be5cd8a1c1e4ff7f1fc55bf97 WatchSource:0}: Error finding container 3f31b6cba0833b20de60259cce82469ea204261be5cd8a1c1e4ff7f1fc55bf97: Status 404 returned error can't find the container with id 3f31b6cba0833b20de60259cce82469ea204261be5cd8a1c1e4ff7f1fc55bf97
Nov 02 13:25:31.898536 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.898499    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2" event={"ID":"517c3f5e-ced3-4482-a1a8-54d37cd2d75f","Type":"ContainerStarted","Data":"e5eefbd24616c6f6bf9e7793ae19a75907259844350eb67e9cb96ec1cd6de863"}
Nov 02 13:25:31.898732 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.898558    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2" event={"ID":"517c3f5e-ced3-4482-a1a8-54d37cd2d75f","Type":"ContainerStarted","Data":"3f31b6cba0833b20de60259cce82469ea204261be5cd8a1c1e4ff7f1fc55bf97"}
Nov 02 13:25:31.898732 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.898631    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"
Nov 02 13:25:31.900358 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.900331    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq" event={"ID":"23c6f60e-aecf-4f24-9d65-fe465b6e1734","Type":"ContainerStarted","Data":"dc1b10b93745d1086e1f65a7f7ff135dea603436fb247fef761d480b14d851e5"}
Nov 02 13:25:31.900484 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.900383    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"
Nov 02 13:25:31.900484 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:31.900405    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq" event={"ID":"23c6f60e-aecf-4f24-9d65-fe465b6e1734","Type":"ContainerStarted","Data":"f9383e03a8074df33d89431447c90c19b23b8a43839f7f933fd24f18c738f708"}
Nov 02 13:25:32.018809 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:32.018757    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2" podStartSLOduration=2.018742269 podStartE2EDuration="2.018742269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:25:32.016955674 +0000 UTC m=+5360.580664118" watchObservedRunningTime="2026-11-02 13:25:32.018742269 +0000 UTC m=+5360.582450716"
Nov 02 13:25:32.141206 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:32.141143    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq" podStartSLOduration=2.141123663 podStartE2EDuration="2.141123663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:25:32.138523547 +0000 UTC m=+5360.702231979" watchObservedRunningTime="2026-11-02 13:25:32.141123663 +0000 UTC m=+5360.704832072"
Nov 02 13:25:42.048987 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:42.048932    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"]
Nov 02 13:25:42.099252 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:42.099224    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7c1e6f1-27lgv"]
Nov 02 13:25:42.908013 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:42.907963    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"
Nov 02 13:25:42.908233 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:42.908047    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"
Nov 02 13:25:43.580954 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:43.580923    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb6e0e22-63ed-491d-9c58-18706c017b89" path="/var/lib/kubelet/pods/cb6e0e22-63ed-491d-9c58-18706c017b89/volumes"
Nov 02 13:25:57.756113 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:57.756080    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-z7qfl_5c2d3d77-9d5c-457e-b93b-c94cf304e188/payload-processing/0.log"
Nov 02 13:25:57.976918 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:57.976891    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-z7qfl_5c2d3d77-9d5c-457e-b93b-c94cf304e188/payload-processing/0.log"
Nov 02 13:25:58.198726 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:25:58.198651    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2_517c3f5e-ced3-4482-a1a8-54d37cd2d75f/payload-processing/0.log"
Nov 02 13:25:58.291157 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:25:58.291122    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:26:08.011606 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:26:08.011570    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:26:12.274884 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:26:12.274860    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:26:12.291439 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:26:12.291417    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:26:12.321302 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:26:12.321278    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:26:12.338182 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:26:12.338151    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:26:28.291229 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:26:28.291195    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:26:52.001347 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:26:52.001310    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:26:58.291922 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:26:58.291844    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:26:59.151368 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:26:59.151339    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds_704f2bb8-c222-405d-985e-b471e50146d3/payload-processing/0.log"
Nov 02 13:26:59.437876 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:26:59.437795    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-z7qfl_5c2d3d77-9d5c-457e-b93b-c94cf304e188/payload-processing/0.log"
Nov 02 13:26:59.663881 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:26:59.663856    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-z7qfl_5c2d3d77-9d5c-457e-b93b-c94cf304e188/payload-processing/0.log"
Nov 02 13:26:59.881919 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:26:59.881890    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2_517c3f5e-ced3-4482-a1a8-54d37cd2d75f/payload-processing/0.log"
Nov 02 13:27:00.089499 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:00.089462    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2_517c3f5e-ced3-4482-a1a8-54d37cd2d75f/payload-processing/0.log"
Nov 02 13:27:14.863837 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:14.863804    4464 scope.go:122] "RemoveContainer" containerID="8a281fafe22749d53fa5fad91c253866eac48077ff9dc5796eec004593344763"
Nov 02 13:27:28.296913 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:27:28.296873    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:27:43.417994 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:43.417941    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"]
Nov 02 13:27:43.422088 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:43.422066    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"
Nov 02 13:27:43.527301 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:43.527272    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:27:43.527480 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:43.527306    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:27:43.527480 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:43.527356    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:27:43.527480 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:43.527371    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:27:43.529739 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:43.529720    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"]
Nov 02 13:27:43.530970 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:43.530953    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j59p\" (UniqueName: \"kubernetes.io/projected/539ba697-a2eb-4f4d-b430-3347d0645c23-kube-api-access-5j59p\") pod \"maas-api-revoke-sub-af65f8a9-llxth\" (UID: \"539ba697-a2eb-4f4d-b430-3347d0645c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"
Nov 02 13:27:43.531046 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:43.531019    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539ba697-a2eb-4f4d-b430-3347d0645c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af65f8a9-llxth\" (UID: \"539ba697-a2eb-4f4d-b430-3347d0645c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"
Nov 02 13:27:43.531150 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:43.531137    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539ba697-a2eb-4f4d-b430-3347d0645c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af65f8a9-llxth\" (UID: \"539ba697-a2eb-4f4d-b430-3347d0645c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"
Nov 02 13:27:43.632468 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:43.632434    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j59p\" (UniqueName: \"kubernetes.io/projected/539ba697-a2eb-4f4d-b430-3347d0645c23-kube-api-access-5j59p\") pod \"maas-api-revoke-sub-af65f8a9-llxth\" (UID: \"539ba697-a2eb-4f4d-b430-3347d0645c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"
Nov 02 13:27:43.632660 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:43.632475    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539ba697-a2eb-4f4d-b430-3347d0645c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af65f8a9-llxth\" (UID: \"539ba697-a2eb-4f4d-b430-3347d0645c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"
Nov 02 13:27:43.632660 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:43.632539    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539ba697-a2eb-4f4d-b430-3347d0645c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af65f8a9-llxth\" (UID: \"539ba697-a2eb-4f4d-b430-3347d0645c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"
Nov 02 13:27:43.633220 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:43.633196    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539ba697-a2eb-4f4d-b430-3347d0645c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af65f8a9-llxth\" (UID: \"539ba697-a2eb-4f4d-b430-3347d0645c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"
Nov 02 13:27:43.634873 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:43.634833    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539ba697-a2eb-4f4d-b430-3347d0645c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af65f8a9-llxth\" (UID: \"539ba697-a2eb-4f4d-b430-3347d0645c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"
Nov 02 13:27:43.696510 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:43.696444    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j59p\" (UniqueName: \"kubernetes.io/projected/539ba697-a2eb-4f4d-b430-3347d0645c23-kube-api-access-5j59p\") pod \"maas-api-revoke-sub-af65f8a9-llxth\" (UID: \"539ba697-a2eb-4f4d-b430-3347d0645c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"
Nov 02 13:27:43.735532 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:43.735506    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"
Nov 02 13:27:44.108343 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:44.108320    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"]
Nov 02 13:27:44.109126 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:27:44.109100    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539ba697_a2eb_4f4d_b430_3347d0645c23.slice/crio-be47a7d59e5ea90c6b6bc501865136f82c562c68fdfc1b90a1926cbda67af122 WatchSource:0}: Error finding container be47a7d59e5ea90c6b6bc501865136f82c562c68fdfc1b90a1926cbda67af122: Status 404 returned error can't find the container with id be47a7d59e5ea90c6b6bc501865136f82c562c68fdfc1b90a1926cbda67af122
Nov 02 13:27:44.588435 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:44.588398    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth" event={"ID":"539ba697-a2eb-4f4d-b430-3347d0645c23","Type":"ContainerStarted","Data":"f28d01210a2b45c7cf435241ac4cf6438f58c69ecb8906e12a877a763fd6fae6"}
Nov 02 13:27:44.588804 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:44.588460    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth" event={"ID":"539ba697-a2eb-4f4d-b430-3347d0645c23","Type":"ContainerStarted","Data":"be47a7d59e5ea90c6b6bc501865136f82c562c68fdfc1b90a1926cbda67af122"}
Nov 02 13:27:44.711295 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:44.711197    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth" podStartSLOduration=1.71118301 podStartE2EDuration="1.71118301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:27:44.708235629 +0000 UTC m=+5493.271944038" watchObservedRunningTime="2026-11-02 13:27:44.71118301 +0000 UTC m=+5493.274891419"
Nov 02 13:27:45.594365 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:45.594327    4464 generic.go:356] "Generic (PLEG): container finished" podID="539ba697-a2eb-4f4d-b430-3347d0645c23" containerID="f28d01210a2b45c7cf435241ac4cf6438f58c69ecb8906e12a877a763fd6fae6" exitCode=0
Nov 02 13:27:45.594779 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:45.594412    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth" event={"ID":"539ba697-a2eb-4f4d-b430-3347d0645c23","Type":"ContainerDied","Data":"f28d01210a2b45c7cf435241ac4cf6438f58c69ecb8906e12a877a763fd6fae6"}
Nov 02 13:27:46.735905 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:46.735880    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"
Nov 02 13:27:46.864135 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:46.864068    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/539ba697-a2eb-4f4d-b430-3347d0645c23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539ba697-a2eb-4f4d-b430-3347d0645c23-maas-api-openshift-ca\") pod \"539ba697-a2eb-4f4d-b430-3347d0645c23\" (UID: \"539ba697-a2eb-4f4d-b430-3347d0645c23\") "
Nov 02 13:27:46.864288 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:46.864146    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/539ba697-a2eb-4f4d-b430-3347d0645c23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539ba697-a2eb-4f4d-b430-3347d0645c23-maas-api-xks-ca\") pod \"539ba697-a2eb-4f4d-b430-3347d0645c23\" (UID: \"539ba697-a2eb-4f4d-b430-3347d0645c23\") "
Nov 02 13:27:46.864288 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:46.864191    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/539ba697-a2eb-4f4d-b430-3347d0645c23-kube-api-access-5j59p\" (UniqueName: \"kubernetes.io/projected/539ba697-a2eb-4f4d-b430-3347d0645c23-kube-api-access-5j59p\") pod \"539ba697-a2eb-4f4d-b430-3347d0645c23\" (UID: \"539ba697-a2eb-4f4d-b430-3347d0645c23\") "
Nov 02 13:27:46.864516 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:46.864488    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/539ba697-a2eb-4f4d-b430-3347d0645c23-maas-api-openshift-ca" pod "539ba697-a2eb-4f4d-b430-3347d0645c23" (UID: "539ba697-a2eb-4f4d-b430-3347d0645c23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:27:46.866254 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:46.866227    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/539ba697-a2eb-4f4d-b430-3347d0645c23-maas-api-xks-ca" pod "539ba697-a2eb-4f4d-b430-3347d0645c23" (UID: "539ba697-a2eb-4f4d-b430-3347d0645c23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:27:46.866351 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:46.866283    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/539ba697-a2eb-4f4d-b430-3347d0645c23-kube-api-access-5j59p" pod "539ba697-a2eb-4f4d-b430-3347d0645c23" (UID: "539ba697-a2eb-4f4d-b430-3347d0645c23"). InnerVolumeSpecName "kube-api-access-5j59p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:27:46.965951 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:46.965921    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j59p\" (UniqueName: \"kubernetes.io/projected/539ba697-a2eb-4f4d-b430-3347d0645c23-kube-api-access-5j59p\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:27:46.965951 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:46.965949    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539ba697-a2eb-4f4d-b430-3347d0645c23-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:27:46.966159 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:46.965959    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539ba697-a2eb-4f4d-b430-3347d0645c23-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:27:47.179057 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:47.178967    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:27:47.606468 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:47.606441    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"
Nov 02 13:27:47.606602 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:47.606466    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth" event={"ID":"539ba697-a2eb-4f4d-b430-3347d0645c23","Type":"ContainerDied","Data":"be47a7d59e5ea90c6b6bc501865136f82c562c68fdfc1b90a1926cbda67af122"}
Nov 02 13:27:47.606602 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:27:47.606510    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be47a7d59e5ea90c6b6bc501865136f82c562c68fdfc1b90a1926cbda67af122"
Nov 02 13:27:58.291010 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:27:58.290954    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:28:05.138544 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.138511    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"]
Nov 02 13:28:05.139059 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.138715    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds" podUID="704f2bb8-c222-405d-985e-b471e50146d3" containerName="payload-processing" containerID="cri-o://57640783b70350c5de412b9fac5a54f2de8b853c5b87be61ef8b134658318d1b" gracePeriod=30
Nov 02 13:28:05.195072 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.195038    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"]
Nov 02 13:28:05.195309 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.195268    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w" podUID="77fca781-cb8d-4524-8d83-16ccd4ced2f1" containerName="payload-pre-processing" containerID="cri-o://e1ba458984371348e6be9dcdc452118a7cd87098281b07927694fa828fbbccd4" gracePeriod=30
Nov 02 13:28:05.460084 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.460054    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"
Nov 02 13:28:05.483669 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.483645    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"
Nov 02 13:28:05.551275 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.551253    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/704f2bb8-c222-405d-985e-b471e50146d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704f2bb8-c222-405d-985e-b471e50146d3-plugins-config-volume\") pod \"704f2bb8-c222-405d-985e-b471e50146d3\" (UID: \"704f2bb8-c222-405d-985e-b471e50146d3\") "
Nov 02 13:28:05.551398 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.551376    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704f2bb8-c222-405d-985e-b471e50146d3-kube-api-access-c2wz9\" (UniqueName: \"kubernetes.io/projected/704f2bb8-c222-405d-985e-b471e50146d3-kube-api-access-c2wz9\") pod \"704f2bb8-c222-405d-985e-b471e50146d3\" (UID: \"704f2bb8-c222-405d-985e-b471e50146d3\") "
Nov 02 13:28:05.551662 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.551639    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704f2bb8-c222-405d-985e-b471e50146d3-plugins-config-volume" pod "704f2bb8-c222-405d-985e-b471e50146d3" (UID: "704f2bb8-c222-405d-985e-b471e50146d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:28:05.553337 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.553318    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704f2bb8-c222-405d-985e-b471e50146d3-kube-api-access-c2wz9" pod "704f2bb8-c222-405d-985e-b471e50146d3" (UID: "704f2bb8-c222-405d-985e-b471e50146d3"). InnerVolumeSpecName "kube-api-access-c2wz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:28:05.652682 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.652606    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77fca781-cb8d-4524-8d83-16ccd4ced2f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77fca781-cb8d-4524-8d83-16ccd4ced2f1-plugins-config-volume\") pod \"77fca781-cb8d-4524-8d83-16ccd4ced2f1\" (UID: \"77fca781-cb8d-4524-8d83-16ccd4ced2f1\") "
Nov 02 13:28:05.652827 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.652763    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77fca781-cb8d-4524-8d83-16ccd4ced2f1-kube-api-access-hhckf\" (UniqueName: \"kubernetes.io/projected/77fca781-cb8d-4524-8d83-16ccd4ced2f1-kube-api-access-hhckf\") pod \"77fca781-cb8d-4524-8d83-16ccd4ced2f1\" (UID: \"77fca781-cb8d-4524-8d83-16ccd4ced2f1\") "
Nov 02 13:28:05.653176 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.653145    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77fca781-cb8d-4524-8d83-16ccd4ced2f1-plugins-config-volume" pod "77fca781-cb8d-4524-8d83-16ccd4ced2f1" (UID: "77fca781-cb8d-4524-8d83-16ccd4ced2f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:28:05.653410 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.653243    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2wz9\" (UniqueName: \"kubernetes.io/projected/704f2bb8-c222-405d-985e-b471e50146d3-kube-api-access-c2wz9\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:28:05.653410 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.653275    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77fca781-cb8d-4524-8d83-16ccd4ced2f1-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:28:05.653410 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.653292    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704f2bb8-c222-405d-985e-b471e50146d3-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:28:05.655420 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.655393    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77fca781-cb8d-4524-8d83-16ccd4ced2f1-kube-api-access-hhckf" pod "77fca781-cb8d-4524-8d83-16ccd4ced2f1" (UID: "77fca781-cb8d-4524-8d83-16ccd4ced2f1"). InnerVolumeSpecName "kube-api-access-hhckf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:28:05.715124 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.715075    4464 generic.go:356] "Generic (PLEG): container finished" podID="77fca781-cb8d-4524-8d83-16ccd4ced2f1" containerID="e1ba458984371348e6be9dcdc452118a7cd87098281b07927694fa828fbbccd4" exitCode=0
Nov 02 13:28:05.715124 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.715121    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"
Nov 02 13:28:05.715392 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.715164    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w" event={"ID":"77fca781-cb8d-4524-8d83-16ccd4ced2f1","Type":"ContainerDied","Data":"e1ba458984371348e6be9dcdc452118a7cd87098281b07927694fa828fbbccd4"}
Nov 02 13:28:05.715392 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.715229    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w" event={"ID":"77fca781-cb8d-4524-8d83-16ccd4ced2f1","Type":"ContainerDied","Data":"f0d429b9d46cc2896cfc9229371b373b338875f388c85530692dfb72e1f3cd54"}
Nov 02 13:28:05.715392 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.715260    4464 scope.go:122] "RemoveContainer" containerID="e1ba458984371348e6be9dcdc452118a7cd87098281b07927694fa828fbbccd4"
Nov 02 13:28:05.716753 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.716733    4464 generic.go:356] "Generic (PLEG): container finished" podID="704f2bb8-c222-405d-985e-b471e50146d3" containerID="57640783b70350c5de412b9fac5a54f2de8b853c5b87be61ef8b134658318d1b" exitCode=0
Nov 02 13:28:05.716871 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.716788    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"
Nov 02 13:28:05.716871 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.716819    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds" event={"ID":"704f2bb8-c222-405d-985e-b471e50146d3","Type":"ContainerDied","Data":"57640783b70350c5de412b9fac5a54f2de8b853c5b87be61ef8b134658318d1b"}
Nov 02 13:28:05.716871 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.716859    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds" event={"ID":"704f2bb8-c222-405d-985e-b471e50146d3","Type":"ContainerDied","Data":"491bd4517f0323146bf0a0eb26c0b86747fc610ccdb4f778ee88e871044ee61e"}
Nov 02 13:28:05.726401 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.726381    4464 scope.go:122] "RemoveContainer" containerID="e1ba458984371348e6be9dcdc452118a7cd87098281b07927694fa828fbbccd4"
Nov 02 13:28:05.726686 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:28:05.726665    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ba458984371348e6be9dcdc452118a7cd87098281b07927694fa828fbbccd4\": container with ID starting with e1ba458984371348e6be9dcdc452118a7cd87098281b07927694fa828fbbccd4 not found: ID does not exist" containerID="e1ba458984371348e6be9dcdc452118a7cd87098281b07927694fa828fbbccd4"
Nov 02 13:28:05.726785 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.726693    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ba458984371348e6be9dcdc452118a7cd87098281b07927694fa828fbbccd4"} err="failed to get container status \"e1ba458984371348e6be9dcdc452118a7cd87098281b07927694fa828fbbccd4\": rpc error: code = NotFound desc = could not find container \"e1ba458984371348e6be9dcdc452118a7cd87098281b07927694fa828fbbccd4\": container with ID starting with e1ba458984371348e6be9dcdc452118a7cd87098281b07927694fa828fbbccd4 not found: ID does not exist"
Nov 02 13:28:05.726785 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.726711    4464 scope.go:122] "RemoveContainer" containerID="57640783b70350c5de412b9fac5a54f2de8b853c5b87be61ef8b134658318d1b"
Nov 02 13:28:05.735394 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.735372    4464 scope.go:122] "RemoveContainer" containerID="57640783b70350c5de412b9fac5a54f2de8b853c5b87be61ef8b134658318d1b"
Nov 02 13:28:05.735653 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:28:05.735633    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57640783b70350c5de412b9fac5a54f2de8b853c5b87be61ef8b134658318d1b\": container with ID starting with 57640783b70350c5de412b9fac5a54f2de8b853c5b87be61ef8b134658318d1b not found: ID does not exist" containerID="57640783b70350c5de412b9fac5a54f2de8b853c5b87be61ef8b134658318d1b"
Nov 02 13:28:05.735719 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.735660    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57640783b70350c5de412b9fac5a54f2de8b853c5b87be61ef8b134658318d1b"} err="failed to get container status \"57640783b70350c5de412b9fac5a54f2de8b853c5b87be61ef8b134658318d1b\": rpc error: code = NotFound desc = could not find container \"57640783b70350c5de412b9fac5a54f2de8b853c5b87be61ef8b134658318d1b\": container with ID starting with 57640783b70350c5de412b9fac5a54f2de8b853c5b87be61ef8b134658318d1b not found: ID does not exist"
Nov 02 13:28:05.753885 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.753860    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhckf\" (UniqueName: \"kubernetes.io/projected/77fca781-cb8d-4524-8d83-16ccd4ced2f1-kube-api-access-hhckf\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:28:05.883054 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.883027    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"]
Nov 02 13:28:05.887509 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.887481    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1e6e34-7bdd688bb6-8bwds"]
Nov 02 13:28:05.999005 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:05.998954    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"]
Nov 02 13:28:06.001075 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:06.001053    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1e6e34-6cf4c5b4bc-gg55w"]
Nov 02 13:28:07.580405 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:07.580371    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="704f2bb8-c222-405d-985e-b471e50146d3" path="/var/lib/kubelet/pods/704f2bb8-c222-405d-985e-b471e50146d3/volumes"
Nov 02 13:28:07.580774 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:07.580762    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77fca781-cb8d-4524-8d83-16ccd4ced2f1" path="/var/lib/kubelet/pods/77fca781-cb8d-4524-8d83-16ccd4ced2f1/volumes"
Nov 02 13:28:09.403776 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:09.403740    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:28:12.049234 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:12.049200    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"]
Nov 02 13:28:12.101356 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:12.101318    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wc2m"]
Nov 02 13:28:13.580860 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:13.580823    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70d7e5d5-5bb2-4eca-867f-f63a088fdd82" path="/var/lib/kubelet/pods/70d7e5d5-5bb2-4eca-867f-f63a088fdd82/volumes"
Nov 02 13:28:13.658460 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:13.658429    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"]
Nov 02 13:28:13.658647 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:13.658630    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z" podUID="704b4f8b-49a9-4b4f-b6bc-81d48164343c" containerName="istio-proxy" containerID="cri-o://c388057c7e09656da966e7dc54a7f4d8be47166f44183e7c71299a6db92af5d8" gracePeriod=30
Nov 02 13:28:14.884664 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:14.884633    4464 scope.go:122] "RemoveContainer" containerID="ff7b7fa14b813230cb4bde90efbdf77c9501392465938e3477f37427462e5d59"
Nov 02 13:28:18.784284 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.784254    4464 generic.go:356] "Generic (PLEG): container finished" podID="704b4f8b-49a9-4b4f-b6bc-81d48164343c" containerID="c388057c7e09656da966e7dc54a7f4d8be47166f44183e7c71299a6db92af5d8" exitCode=0
Nov 02 13:28:18.784712 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.784326    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z" event={"ID":"704b4f8b-49a9-4b4f-b6bc-81d48164343c","Type":"ContainerDied","Data":"c388057c7e09656da966e7dc54a7f4d8be47166f44183e7c71299a6db92af5d8"}
Nov 02 13:28:18.784712 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.784443    4464 scope.go:122] "RemoveContainer" containerID="a46433bcd0d378e8ed6b48154b4cd1fdaa6830fa4ca2f712aba82856356ec319"
Nov 02 13:28:18.940039 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.939963    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:28:18.996884 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.996854    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-envoy\") pod \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") "
Nov 02 13:28:18.997103 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.996902    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-podinfo\") pod \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") "
Nov 02 13:28:18.997103 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.996944    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istiod-ca-cert\") pod \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") "
Nov 02 13:28:18.997103 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.997002    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-workload-socket\") pod \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") "
Nov 02 13:28:18.997103 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.997054    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-data\") pod \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") "
Nov 02 13:28:18.997382 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.997139    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-token\" (UniqueName: \"kubernetes.io/projected/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-token\") pod \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") "
Nov 02 13:28:18.997382 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.997178    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704b4f8b-49a9-4b4f-b6bc-81d48164343c-kube-api-access-f8tqz\" (UniqueName: \"kubernetes.io/projected/704b4f8b-49a9-4b4f-b6bc-81d48164343c-kube-api-access-f8tqz\") pod \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") "
Nov 02 13:28:18.997382 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.997226    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-workload-certs\") pod \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") "
Nov 02 13:28:18.997382 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.997298    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-credential-socket\") pod \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\" (UID: \"704b4f8b-49a9-4b4f-b6bc-81d48164343c\") "
Nov 02 13:28:18.997382 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.997350    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-workload-socket" pod "704b4f8b-49a9-4b4f-b6bc-81d48164343c" (UID: "704b4f8b-49a9-4b4f-b6bc-81d48164343c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:28:18.997666 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.997465    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-data" pod "704b4f8b-49a9-4b4f-b6bc-81d48164343c" (UID: "704b4f8b-49a9-4b4f-b6bc-81d48164343c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:28:18.997666 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.997388    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istiod-ca-cert" pod "704b4f8b-49a9-4b4f-b6bc-81d48164343c" (UID: "704b4f8b-49a9-4b4f-b6bc-81d48164343c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:28:18.997666 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.997584    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-workload-certs" pod "704b4f8b-49a9-4b4f-b6bc-81d48164343c" (UID: "704b4f8b-49a9-4b4f-b6bc-81d48164343c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:28:18.997844 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.997681    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-credential-socket" pod "704b4f8b-49a9-4b4f-b6bc-81d48164343c" (UID: "704b4f8b-49a9-4b4f-b6bc-81d48164343c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:28:18.997844 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.997695    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:28:18.997844 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.997716    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:28:18.997844 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.997732    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:28:18.997844 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.997745    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:28:18.999256 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.999223    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-token" pod "704b4f8b-49a9-4b4f-b6bc-81d48164343c" (UID: "704b4f8b-49a9-4b4f-b6bc-81d48164343c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:28:18.999256 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.999243    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-envoy" pod "704b4f8b-49a9-4b4f-b6bc-81d48164343c" (UID: "704b4f8b-49a9-4b4f-b6bc-81d48164343c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:28:18.999443 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.999286    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-podinfo" pod "704b4f8b-49a9-4b4f-b6bc-81d48164343c" (UID: "704b4f8b-49a9-4b4f-b6bc-81d48164343c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:28:18.999509 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:18.999438    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704b4f8b-49a9-4b4f-b6bc-81d48164343c-kube-api-access-f8tqz" pod "704b4f8b-49a9-4b4f-b6bc-81d48164343c" (UID: "704b4f8b-49a9-4b4f-b6bc-81d48164343c"). InnerVolumeSpecName "kube-api-access-f8tqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:28:19.099096 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:19.099064    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:28:19.099096 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:19.099098    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8tqz\" (UniqueName: \"kubernetes.io/projected/704b4f8b-49a9-4b4f-b6bc-81d48164343c-kube-api-access-f8tqz\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:28:19.099096 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:19.099113    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:28:19.099358 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:19.099126    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:28:19.099358 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:19.099138    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/704b4f8b-49a9-4b4f-b6bc-81d48164343c-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:28:19.790920 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:19.790893    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"
Nov 02 13:28:19.791374 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:19.790924    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z" event={"ID":"704b4f8b-49a9-4b4f-b6bc-81d48164343c","Type":"ContainerDied","Data":"b5d0ca03f8783b75d35aa8b9e333f69bee3ae2a5ca64d9f8c235dd2530477c6c"}
Nov 02 13:28:19.791374 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:19.791006    4464 scope.go:122] "RemoveContainer" containerID="c388057c7e09656da966e7dc54a7f4d8be47166f44183e7c71299a6db92af5d8"
Nov 02 13:28:20.009309 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:20.009275    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"]
Nov 02 13:28:20.060767 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:20.060733    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1e6e34-openshift-default-6f5755c8dc-fsr7z"]
Nov 02 13:28:21.581604 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:21.581575    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="704b4f8b-49a9-4b4f-b6bc-81d48164343c" path="/var/lib/kubelet/pods/704b4f8b-49a9-4b4f-b6bc-81d48164343c/volumes"
Nov 02 13:28:28.291600 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:28:28.291525    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:28:35.675307 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:35.675270    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"]
Nov 02 13:28:35.675713 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:35.675444    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2" podUID="517c3f5e-ced3-4482-a1a8-54d37cd2d75f" containerName="payload-processing" containerID="cri-o://e5eefbd24616c6f6bf9e7793ae19a75907259844350eb67e9cb96ec1cd6de863" gracePeriod=30
Nov 02 13:28:35.782377 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:35.782344    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"]
Nov 02 13:28:35.782606 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:35.782581    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq" podUID="23c6f60e-aecf-4f24-9d65-fe465b6e1734" containerName="payload-pre-processing" containerID="cri-o://dc1b10b93745d1086e1f65a7f7ff135dea603436fb247fef761d480b14d851e5" gracePeriod=30
Nov 02 13:28:35.878296 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:35.878261    4464 generic.go:356] "Generic (PLEG): container finished" podID="517c3f5e-ced3-4482-a1a8-54d37cd2d75f" containerID="e5eefbd24616c6f6bf9e7793ae19a75907259844350eb67e9cb96ec1cd6de863" exitCode=0
Nov 02 13:28:35.878470 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:35.878331    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2" event={"ID":"517c3f5e-ced3-4482-a1a8-54d37cd2d75f","Type":"ContainerDied","Data":"e5eefbd24616c6f6bf9e7793ae19a75907259844350eb67e9cb96ec1cd6de863"}
Nov 02 13:28:35.953383 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:35.953355    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"
Nov 02 13:28:36.032728 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.032705    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"
Nov 02 13:28:36.072507 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.072066    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/517c3f5e-ced3-4482-a1a8-54d37cd2d75f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/517c3f5e-ced3-4482-a1a8-54d37cd2d75f-plugins-config-volume\") pod \"517c3f5e-ced3-4482-a1a8-54d37cd2d75f\" (UID: \"517c3f5e-ced3-4482-a1a8-54d37cd2d75f\") "
Nov 02 13:28:36.072507 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.072147    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/517c3f5e-ced3-4482-a1a8-54d37cd2d75f-kube-api-access-jn7zf\" (UniqueName: \"kubernetes.io/projected/517c3f5e-ced3-4482-a1a8-54d37cd2d75f-kube-api-access-jn7zf\") pod \"517c3f5e-ced3-4482-a1a8-54d37cd2d75f\" (UID: \"517c3f5e-ced3-4482-a1a8-54d37cd2d75f\") "
Nov 02 13:28:36.072711 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.072666    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/517c3f5e-ced3-4482-a1a8-54d37cd2d75f-plugins-config-volume" pod "517c3f5e-ced3-4482-a1a8-54d37cd2d75f" (UID: "517c3f5e-ced3-4482-a1a8-54d37cd2d75f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:28:36.075077 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.075054    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517c3f5e-ced3-4482-a1a8-54d37cd2d75f-kube-api-access-jn7zf" pod "517c3f5e-ced3-4482-a1a8-54d37cd2d75f" (UID: "517c3f5e-ced3-4482-a1a8-54d37cd2d75f"). InnerVolumeSpecName "kube-api-access-jn7zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:28:36.181330 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.181110    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c6f60e-aecf-4f24-9d65-fe465b6e1734-kube-api-access-hq4ck\" (UniqueName: \"kubernetes.io/projected/23c6f60e-aecf-4f24-9d65-fe465b6e1734-kube-api-access-hq4ck\") pod \"23c6f60e-aecf-4f24-9d65-fe465b6e1734\" (UID: \"23c6f60e-aecf-4f24-9d65-fe465b6e1734\") "
Nov 02 13:28:36.185553 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.181748    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c6f60e-aecf-4f24-9d65-fe465b6e1734-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c6f60e-aecf-4f24-9d65-fe465b6e1734-plugins-config-volume\") pod \"23c6f60e-aecf-4f24-9d65-fe465b6e1734\" (UID: \"23c6f60e-aecf-4f24-9d65-fe465b6e1734\") "
Nov 02 13:28:36.185553 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.182573    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn7zf\" (UniqueName: \"kubernetes.io/projected/517c3f5e-ced3-4482-a1a8-54d37cd2d75f-kube-api-access-jn7zf\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:28:36.185553 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.182603    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/517c3f5e-ced3-4482-a1a8-54d37cd2d75f-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:28:36.185553 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.182175    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c6f60e-aecf-4f24-9d65-fe465b6e1734-plugins-config-volume" pod "23c6f60e-aecf-4f24-9d65-fe465b6e1734" (UID: "23c6f60e-aecf-4f24-9d65-fe465b6e1734"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:28:36.187021 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.186364    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c6f60e-aecf-4f24-9d65-fe465b6e1734-kube-api-access-hq4ck" pod "23c6f60e-aecf-4f24-9d65-fe465b6e1734" (UID: "23c6f60e-aecf-4f24-9d65-fe465b6e1734"). InnerVolumeSpecName "kube-api-access-hq4ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:28:36.283248 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.283214    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq4ck\" (UniqueName: \"kubernetes.io/projected/23c6f60e-aecf-4f24-9d65-fe465b6e1734-kube-api-access-hq4ck\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:28:36.283248 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.283244    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c6f60e-aecf-4f24-9d65-fe465b6e1734-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:28:36.884109 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.884073    4464 generic.go:356] "Generic (PLEG): container finished" podID="23c6f60e-aecf-4f24-9d65-fe465b6e1734" containerID="dc1b10b93745d1086e1f65a7f7ff135dea603436fb247fef761d480b14d851e5" exitCode=0
Nov 02 13:28:36.884545 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.884127    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"
Nov 02 13:28:36.884545 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.884162    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq" event={"ID":"23c6f60e-aecf-4f24-9d65-fe465b6e1734","Type":"ContainerDied","Data":"dc1b10b93745d1086e1f65a7f7ff135dea603436fb247fef761d480b14d851e5"}
Nov 02 13:28:36.884545 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.884228    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq" event={"ID":"23c6f60e-aecf-4f24-9d65-fe465b6e1734","Type":"ContainerDied","Data":"f9383e03a8074df33d89431447c90c19b23b8a43839f7f933fd24f18c738f708"}
Nov 02 13:28:36.884545 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.884263    4464 scope.go:122] "RemoveContainer" containerID="dc1b10b93745d1086e1f65a7f7ff135dea603436fb247fef761d480b14d851e5"
Nov 02 13:28:36.885602 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.885581    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2" event={"ID":"517c3f5e-ced3-4482-a1a8-54d37cd2d75f","Type":"ContainerDied","Data":"3f31b6cba0833b20de60259cce82469ea204261be5cd8a1c1e4ff7f1fc55bf97"}
Nov 02 13:28:36.885692 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.885601    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"
Nov 02 13:28:36.893825 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.893809    4464 scope.go:122] "RemoveContainer" containerID="dc1b10b93745d1086e1f65a7f7ff135dea603436fb247fef761d480b14d851e5"
Nov 02 13:28:36.894108 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:28:36.894090    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1b10b93745d1086e1f65a7f7ff135dea603436fb247fef761d480b14d851e5\": container with ID starting with dc1b10b93745d1086e1f65a7f7ff135dea603436fb247fef761d480b14d851e5 not found: ID does not exist" containerID="dc1b10b93745d1086e1f65a7f7ff135dea603436fb247fef761d480b14d851e5"
Nov 02 13:28:36.894177 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.894114    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1b10b93745d1086e1f65a7f7ff135dea603436fb247fef761d480b14d851e5"} err="failed to get container status \"dc1b10b93745d1086e1f65a7f7ff135dea603436fb247fef761d480b14d851e5\": rpc error: code = NotFound desc = could not find container \"dc1b10b93745d1086e1f65a7f7ff135dea603436fb247fef761d480b14d851e5\": container with ID starting with dc1b10b93745d1086e1f65a7f7ff135dea603436fb247fef761d480b14d851e5 not found: ID does not exist"
Nov 02 13:28:36.894177 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:36.894128    4464 scope.go:122] "RemoveContainer" containerID="e5eefbd24616c6f6bf9e7793ae19a75907259844350eb67e9cb96ec1cd6de863"
Nov 02 13:28:37.057203 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:37.053904    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"]
Nov 02 13:28:37.058104 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:37.058077    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-565df96f5-n5jfq"]
Nov 02 13:28:37.167741 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:37.167712    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"]
Nov 02 13:28:37.172222 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:37.172201    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-193883-66f8d96c44-z48c2"]
Nov 02 13:28:37.580926 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:37.580895    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c6f60e-aecf-4f24-9d65-fe465b6e1734" path="/var/lib/kubelet/pods/23c6f60e-aecf-4f24-9d65-fe465b6e1734/volumes"
Nov 02 13:28:37.581287 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:37.581274    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="517c3f5e-ced3-4482-a1a8-54d37cd2d75f" path="/var/lib/kubelet/pods/517c3f5e-ced3-4482-a1a8-54d37cd2d75f/volumes"
Nov 02 13:28:39.706705 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:39.706668    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:28:44.142315 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.142282    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-58769f698d-9hnjj"]
Nov 02 13:28:44.142727 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.142710    4464 cpu_manager.go:414] "containerMap: removing container" podUID="517c3f5e-ced3-4482-a1a8-54d37cd2d75f" containerName="payload-processing"
Nov 02 13:28:44.142727 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.142723    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="517c3f5e-ced3-4482-a1a8-54d37cd2d75f" containerName="payload-processing"
Nov 02 13:28:44.142804 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.142767    4464 cpu_manager.go:414] "containerMap: removing container" podUID="23c6f60e-aecf-4f24-9d65-fe465b6e1734" containerName="payload-pre-processing"
Nov 02 13:28:44.142804 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.142774    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c6f60e-aecf-4f24-9d65-fe465b6e1734" containerName="payload-pre-processing"
Nov 02 13:28:44.142804 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.142783    4464 cpu_manager.go:414] "containerMap: removing container" podUID="704b4f8b-49a9-4b4f-b6bc-81d48164343c" containerName="istio-proxy"
Nov 02 13:28:44.142804 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.142788    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704b4f8b-49a9-4b4f-b6bc-81d48164343c" containerName="istio-proxy"
Nov 02 13:28:44.142946 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.142828    4464 cpu_manager.go:414] "containerMap: removing container" podUID="704f2bb8-c222-405d-985e-b471e50146d3" containerName="payload-processing"
Nov 02 13:28:44.142946 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.142836    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704f2bb8-c222-405d-985e-b471e50146d3" containerName="payload-processing"
Nov 02 13:28:44.142946 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.142846    4464 cpu_manager.go:414] "containerMap: removing container" podUID="77fca781-cb8d-4524-8d83-16ccd4ced2f1" containerName="payload-pre-processing"
Nov 02 13:28:44.142946 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.142851    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77fca781-cb8d-4524-8d83-16ccd4ced2f1" containerName="payload-pre-processing"
Nov 02 13:28:44.142946 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.142876    4464 cpu_manager.go:414] "containerMap: removing container" podUID="539ba697-a2eb-4f4d-b430-3347d0645c23" containerName="revoke-keys"
Nov 02 13:28:44.142946 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.142881    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="539ba697-a2eb-4f4d-b430-3347d0645c23" containerName="revoke-keys"
Nov 02 13:28:44.143289 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.143007    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="704f2bb8-c222-405d-985e-b471e50146d3" containerName="payload-processing"
Nov 02 13:28:44.143289 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.143019    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="77fca781-cb8d-4524-8d83-16ccd4ced2f1" containerName="payload-pre-processing"
Nov 02 13:28:44.143289 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.143031    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="539ba697-a2eb-4f4d-b430-3347d0645c23" containerName="revoke-keys"
Nov 02 13:28:44.143289 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.143046    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="704b4f8b-49a9-4b4f-b6bc-81d48164343c" containerName="istio-proxy"
Nov 02 13:28:44.143289 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.143056    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="517c3f5e-ced3-4482-a1a8-54d37cd2d75f" containerName="payload-processing"
Nov 02 13:28:44.143289 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.143069    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c6f60e-aecf-4f24-9d65-fe465b6e1734" containerName="payload-pre-processing"
Nov 02 13:28:44.147804 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.147789    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj"
Nov 02 13:28:44.156892 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.156862    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwktf\" (UniqueName: \"kubernetes.io/projected/bd7e087d-7840-4930-bc46-e39d15f4b00e-kube-api-access-bwktf\") pod \"payload-pre-processing-58769f698d-9hnjj\" (UID: \"bd7e087d-7840-4930-bc46-e39d15f4b00e\") " pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj"
Nov 02 13:28:44.157018 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.156992    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7e087d-7840-4930-bc46-e39d15f4b00e-plugins-config-volume\") pod \"payload-pre-processing-58769f698d-9hnjj\" (UID: \"bd7e087d-7840-4930-bc46-e39d15f4b00e\") " pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj"
Nov 02 13:28:44.203968 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.203940    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-58769f698d-9hnjj"]
Nov 02 13:28:44.257592 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.257560    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwktf\" (UniqueName: \"kubernetes.io/projected/bd7e087d-7840-4930-bc46-e39d15f4b00e-kube-api-access-bwktf\") pod \"payload-pre-processing-58769f698d-9hnjj\" (UID: \"bd7e087d-7840-4930-bc46-e39d15f4b00e\") " pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj"
Nov 02 13:28:44.257756 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.257619    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7e087d-7840-4930-bc46-e39d15f4b00e-plugins-config-volume\") pod \"payload-pre-processing-58769f698d-9hnjj\" (UID: \"bd7e087d-7840-4930-bc46-e39d15f4b00e\") " pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj"
Nov 02 13:28:44.258271 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.258251    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7e087d-7840-4930-bc46-e39d15f4b00e-plugins-config-volume\") pod \"payload-pre-processing-58769f698d-9hnjj\" (UID: \"bd7e087d-7840-4930-bc46-e39d15f4b00e\") " pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj"
Nov 02 13:28:44.320015 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.319960    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwktf\" (UniqueName: \"kubernetes.io/projected/bd7e087d-7840-4930-bc46-e39d15f4b00e-kube-api-access-bwktf\") pod \"payload-pre-processing-58769f698d-9hnjj\" (UID: \"bd7e087d-7840-4930-bc46-e39d15f4b00e\") " pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj"
Nov 02 13:28:44.460430 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.460344    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj"
Nov 02 13:28:44.833846 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.833818    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-58769f698d-9hnjj"]
Nov 02 13:28:44.835835 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:28:44.835805    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7e087d_7840_4930_bc46_e39d15f4b00e.slice/crio-5b99f663721e294aa8246e5d96e7ef4d6ec4e2ba1f16f53b8f3b4eccc2644d63 WatchSource:0}: Error finding container 5b99f663721e294aa8246e5d96e7ef4d6ec4e2ba1f16f53b8f3b4eccc2644d63: Status 404 returned error can't find the container with id 5b99f663721e294aa8246e5d96e7ef4d6ec4e2ba1f16f53b8f3b4eccc2644d63
Nov 02 13:28:44.933389 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.933363    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj" event={"ID":"bd7e087d-7840-4930-bc46-e39d15f4b00e","Type":"ContainerStarted","Data":"cf02793df04d2691be6c0a38fccdfeddf4733d64d074b0c346725d6e4f737596"}
Nov 02 13:28:44.933502 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.933409    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj" event={"ID":"bd7e087d-7840-4930-bc46-e39d15f4b00e","Type":"ContainerStarted","Data":"5b99f663721e294aa8246e5d96e7ef4d6ec4e2ba1f16f53b8f3b4eccc2644d63"}
Nov 02 13:28:44.933502 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:44.933472    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj"
Nov 02 13:28:45.051554 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:45.051497    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj" podStartSLOduration=1.051481951 podStartE2EDuration="1.051481951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:28:45.051046988 +0000 UTC m=+5553.614755401" watchObservedRunningTime="2026-11-02 13:28:45.051481951 +0000 UTC m=+5553.615190359"
Nov 02 13:28:47.048390 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:47.048352    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"]
Nov 02 13:28:47.099058 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:47.099026    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-xb45d"]
Nov 02 13:28:47.580952 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:47.580919    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7901efe8-9ffe-4931-8185-3112b79085a0" path="/var/lib/kubelet/pods/7901efe8-9ffe-4931-8185-3112b79085a0/volumes"
Nov 02 13:28:55.940791 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:55.940765    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj"
Nov 02 13:28:56.111749 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:56.111715    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"]
Nov 02 13:28:58.077633 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:58.077598    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5d8fcd46d4-q24jg"]
Nov 02 13:28:58.078148 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:58.078128    4464 cpu_manager.go:414] "containerMap: removing container" podUID="704b4f8b-49a9-4b4f-b6bc-81d48164343c" containerName="istio-proxy"
Nov 02 13:28:58.078247 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:58.078151    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704b4f8b-49a9-4b4f-b6bc-81d48164343c" containerName="istio-proxy"
Nov 02 13:28:58.078291 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:58.078271    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="704b4f8b-49a9-4b4f-b6bc-81d48164343c" containerName="istio-proxy"
Nov 02 13:28:58.082604 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:58.082584    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg"
Nov 02 13:28:58.141543 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:58.141507    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5d8fcd46d4-q24jg"]
Nov 02 13:28:58.183440 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:58.183413    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plpts\" (UniqueName: \"kubernetes.io/projected/bde9e4c9-6824-4557-b085-b13066e25ea4-kube-api-access-plpts\") pod \"payload-processing-5d8fcd46d4-q24jg\" (UID: \"bde9e4c9-6824-4557-b085-b13066e25ea4\") " pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg"
Nov 02 13:28:58.183588 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:58.183477    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bde9e4c9-6824-4557-b085-b13066e25ea4-plugins-config-volume\") pod \"payload-processing-5d8fcd46d4-q24jg\" (UID: \"bde9e4c9-6824-4557-b085-b13066e25ea4\") " pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg"
Nov 02 13:28:58.284791 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:58.284758    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bde9e4c9-6824-4557-b085-b13066e25ea4-plugins-config-volume\") pod \"payload-processing-5d8fcd46d4-q24jg\" (UID: \"bde9e4c9-6824-4557-b085-b13066e25ea4\") " pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg"
Nov 02 13:28:58.284971 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:58.284948    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plpts\" (UniqueName: \"kubernetes.io/projected/bde9e4c9-6824-4557-b085-b13066e25ea4-kube-api-access-plpts\") pod \"payload-processing-5d8fcd46d4-q24jg\" (UID: \"bde9e4c9-6824-4557-b085-b13066e25ea4\") " pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg"
Nov 02 13:28:58.285575 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:58.285552    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bde9e4c9-6824-4557-b085-b13066e25ea4-plugins-config-volume\") pod \"payload-processing-5d8fcd46d4-q24jg\" (UID: \"bde9e4c9-6824-4557-b085-b13066e25ea4\") " pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg"
Nov 02 13:28:58.291721 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:28:58.291692    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:28:58.347361 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:58.347307    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plpts\" (UniqueName: \"kubernetes.io/projected/bde9e4c9-6824-4557-b085-b13066e25ea4-kube-api-access-plpts\") pod \"payload-processing-5d8fcd46d4-q24jg\" (UID: \"bde9e4c9-6824-4557-b085-b13066e25ea4\") " pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg"
Nov 02 13:28:58.395698 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:58.395673    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg"
Nov 02 13:28:58.766597 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:58.766572    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5d8fcd46d4-q24jg"]
Nov 02 13:28:58.768105 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:28:58.768076    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde9e4c9_6824_4557_b085_b13066e25ea4.slice/crio-1d160233ff1c0fc2e1d792f54c395794f1d3fdb0a0707d4eb63004c1b3ae7b01 WatchSource:0}: Error finding container 1d160233ff1c0fc2e1d792f54c395794f1d3fdb0a0707d4eb63004c1b3ae7b01: Status 404 returned error can't find the container with id 1d160233ff1c0fc2e1d792f54c395794f1d3fdb0a0707d4eb63004c1b3ae7b01
Nov 02 13:28:59.004365 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:59.004330    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg" event={"ID":"bde9e4c9-6824-4557-b085-b13066e25ea4","Type":"ContainerStarted","Data":"f4abbf03a4f6885dd312cd34107cb7a92c36b51969ecc9a7685276c18f56c94b"}
Nov 02 13:28:59.004568 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:59.004390    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg" event={"ID":"bde9e4c9-6824-4557-b085-b13066e25ea4","Type":"ContainerStarted","Data":"1d160233ff1c0fc2e1d792f54c395794f1d3fdb0a0707d4eb63004c1b3ae7b01"}
Nov 02 13:28:59.004568 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:59.004416    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg"
Nov 02 13:28:59.123106 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:28:59.122999    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg" podStartSLOduration=1.122962406 podStartE2EDuration="1.122962406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:28:59.122576746 +0000 UTC m=+5567.686285157" watchObservedRunningTime="2026-11-02 13:28:59.122962406 +0000 UTC m=+5567.686670819"
Nov 02 13:29:01.112246 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:01.112192    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc" podUID="0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c" containerName="payload-pre-processing" containerID="cri-o://655ff2677b95df3b8230fe110c472ed137f27fb271f41f592adfaabebf3b776c" gracePeriod=25
Nov 02 13:29:01.363226 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:01.363167    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"
Nov 02 13:29:01.415312 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:01.415288    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c-kube-api-access-rwc7d\" (UniqueName: \"kubernetes.io/projected/0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c-kube-api-access-rwc7d\") pod \"0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c\" (UID: \"0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c\") "
Nov 02 13:29:01.415458 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:01.415342    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c-plugins-config-volume\") pod \"0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c\" (UID: \"0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c\") "
Nov 02 13:29:01.415736 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:01.415714    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c-plugins-config-volume" pod "0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c" (UID: "0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:01.417251 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:01.417229    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c-kube-api-access-rwc7d" pod "0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c" (UID: "0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c"). InnerVolumeSpecName "kube-api-access-rwc7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:01.516694 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:01.516662    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwc7d\" (UniqueName: \"kubernetes.io/projected/0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c-kube-api-access-rwc7d\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:01.516694 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:01.516688    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:02.030145 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:02.030111    4464 generic.go:356] "Generic (PLEG): container finished" podID="0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c" containerID="655ff2677b95df3b8230fe110c472ed137f27fb271f41f592adfaabebf3b776c" exitCode=0
Nov 02 13:29:02.030324 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:02.030166    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"
Nov 02 13:29:02.030324 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:02.030189    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc" event={"ID":"0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c","Type":"ContainerDied","Data":"655ff2677b95df3b8230fe110c472ed137f27fb271f41f592adfaabebf3b776c"}
Nov 02 13:29:02.030324 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:02.030254    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc" event={"ID":"0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c","Type":"ContainerDied","Data":"d786a8faaad3c431417c0a4df94a5393c39e888db8de4272b9970acc0c276420"}
Nov 02 13:29:02.030324 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:02.030274    4464 scope.go:122] "RemoveContainer" containerID="655ff2677b95df3b8230fe110c472ed137f27fb271f41f592adfaabebf3b776c"
Nov 02 13:29:02.038877 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:02.038859    4464 scope.go:122] "RemoveContainer" containerID="655ff2677b95df3b8230fe110c472ed137f27fb271f41f592adfaabebf3b776c"
Nov 02 13:29:02.039168 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:29:02.039149    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"655ff2677b95df3b8230fe110c472ed137f27fb271f41f592adfaabebf3b776c\": container with ID starting with 655ff2677b95df3b8230fe110c472ed137f27fb271f41f592adfaabebf3b776c not found: ID does not exist" containerID="655ff2677b95df3b8230fe110c472ed137f27fb271f41f592adfaabebf3b776c"
Nov 02 13:29:02.039221 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:02.039176    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"655ff2677b95df3b8230fe110c472ed137f27fb271f41f592adfaabebf3b776c"} err="failed to get container status \"655ff2677b95df3b8230fe110c472ed137f27fb271f41f592adfaabebf3b776c\": rpc error: code = NotFound desc = could not find container \"655ff2677b95df3b8230fe110c472ed137f27fb271f41f592adfaabebf3b776c\": container with ID starting with 655ff2677b95df3b8230fe110c472ed137f27fb271f41f592adfaabebf3b776c not found: ID does not exist"
Nov 02 13:29:02.197324 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:02.197293    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"]
Nov 02 13:29:02.247895 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:02.247864    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-hf7lc"]
Nov 02 13:29:03.580589 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:03.580555    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c" path="/var/lib/kubelet/pods/0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c/volumes"
Nov 02 13:29:10.013712 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:10.013684    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg"
Nov 02 13:29:10.184260 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:10.184223    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"]
Nov 02 13:29:12.194459 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.194418    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-656dd59d89-ml7g4"]
Nov 02 13:29:12.194916 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.194902    4464 cpu_manager.go:414] "containerMap: removing container" podUID="0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c" containerName="payload-pre-processing"
Nov 02 13:29:12.194996 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.194918    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c" containerName="payload-pre-processing"
Nov 02 13:29:12.195057 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.195009    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc6178f-53db-4bf6-ae1f-3def4e3b9f4c" containerName="payload-pre-processing"
Nov 02 13:29:12.198116 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.198099    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-656dd59d89-ml7g4"
Nov 02 13:29:12.255952 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.255916    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-656dd59d89-ml7g4"]
Nov 02 13:29:12.326425 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.326388    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hkbs\" (UniqueName: \"kubernetes.io/projected/2e12c7b1-dc24-4d4a-9cf4-682f4e82434d-kube-api-access-9hkbs\") pod \"payload-processing-656dd59d89-ml7g4\" (UID: \"2e12c7b1-dc24-4d4a-9cf4-682f4e82434d\") " pod="openshift-ingress/payload-processing-656dd59d89-ml7g4"
Nov 02 13:29:12.326587 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.326503    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e12c7b1-dc24-4d4a-9cf4-682f4e82434d-plugins-config-volume\") pod \"payload-processing-656dd59d89-ml7g4\" (UID: \"2e12c7b1-dc24-4d4a-9cf4-682f4e82434d\") " pod="openshift-ingress/payload-processing-656dd59d89-ml7g4"
Nov 02 13:29:12.343364 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.343332    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-69b46c5574-d6tqb"]
Nov 02 13:29:12.346882 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.346862    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-69b46c5574-d6tqb"
Nov 02 13:29:12.404137 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.404106    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-69b46c5574-d6tqb"]
Nov 02 13:29:12.427720 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.427694    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hkbs\" (UniqueName: \"kubernetes.io/projected/2e12c7b1-dc24-4d4a-9cf4-682f4e82434d-kube-api-access-9hkbs\") pod \"payload-processing-656dd59d89-ml7g4\" (UID: \"2e12c7b1-dc24-4d4a-9cf4-682f4e82434d\") " pod="openshift-ingress/payload-processing-656dd59d89-ml7g4"
Nov 02 13:29:12.427911 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.427751    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vlwz\" (UniqueName: \"kubernetes.io/projected/cd28a3fe-11b1-491e-b7f7-c10276803a4f-kube-api-access-6vlwz\") pod \"payload-pre-processing-69b46c5574-d6tqb\" (UID: \"cd28a3fe-11b1-491e-b7f7-c10276803a4f\") " pod="openshift-ingress/payload-pre-processing-69b46c5574-d6tqb"
Nov 02 13:29:12.427911 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.427847    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd28a3fe-11b1-491e-b7f7-c10276803a4f-plugins-config-volume\") pod \"payload-pre-processing-69b46c5574-d6tqb\" (UID: \"cd28a3fe-11b1-491e-b7f7-c10276803a4f\") " pod="openshift-ingress/payload-pre-processing-69b46c5574-d6tqb"
Nov 02 13:29:12.428086 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.427985    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e12c7b1-dc24-4d4a-9cf4-682f4e82434d-plugins-config-volume\") pod \"payload-processing-656dd59d89-ml7g4\" (UID: \"2e12c7b1-dc24-4d4a-9cf4-682f4e82434d\") " pod="openshift-ingress/payload-processing-656dd59d89-ml7g4"
Nov 02 13:29:12.428864 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.428842    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e12c7b1-dc24-4d4a-9cf4-682f4e82434d-plugins-config-volume\") pod \"payload-processing-656dd59d89-ml7g4\" (UID: \"2e12c7b1-dc24-4d4a-9cf4-682f4e82434d\") " pod="openshift-ingress/payload-processing-656dd59d89-ml7g4"
Nov 02 13:29:12.490061 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.490025    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hkbs\" (UniqueName: \"kubernetes.io/projected/2e12c7b1-dc24-4d4a-9cf4-682f4e82434d-kube-api-access-9hkbs\") pod \"payload-processing-656dd59d89-ml7g4\" (UID: \"2e12c7b1-dc24-4d4a-9cf4-682f4e82434d\") " pod="openshift-ingress/payload-processing-656dd59d89-ml7g4"
Nov 02 13:29:12.510801 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.510777    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-656dd59d89-ml7g4"
Nov 02 13:29:12.529092 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.529065    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vlwz\" (UniqueName: \"kubernetes.io/projected/cd28a3fe-11b1-491e-b7f7-c10276803a4f-kube-api-access-6vlwz\") pod \"payload-pre-processing-69b46c5574-d6tqb\" (UID: \"cd28a3fe-11b1-491e-b7f7-c10276803a4f\") " pod="openshift-ingress/payload-pre-processing-69b46c5574-d6tqb"
Nov 02 13:29:12.529204 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.529108    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd28a3fe-11b1-491e-b7f7-c10276803a4f-plugins-config-volume\") pod \"payload-pre-processing-69b46c5574-d6tqb\" (UID: \"cd28a3fe-11b1-491e-b7f7-c10276803a4f\") " pod="openshift-ingress/payload-pre-processing-69b46c5574-d6tqb"
Nov 02 13:29:12.529842 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.529822    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd28a3fe-11b1-491e-b7f7-c10276803a4f-plugins-config-volume\") pod \"payload-pre-processing-69b46c5574-d6tqb\" (UID: \"cd28a3fe-11b1-491e-b7f7-c10276803a4f\") " pod="openshift-ingress/payload-pre-processing-69b46c5574-d6tqb"
Nov 02 13:29:12.592380 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.592343    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vlwz\" (UniqueName: \"kubernetes.io/projected/cd28a3fe-11b1-491e-b7f7-c10276803a4f-kube-api-access-6vlwz\") pod \"payload-pre-processing-69b46c5574-d6tqb\" (UID: \"cd28a3fe-11b1-491e-b7f7-c10276803a4f\") " pod="openshift-ingress/payload-pre-processing-69b46c5574-d6tqb"
Nov 02 13:29:12.659405 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.659376    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-69b46c5574-d6tqb"
Nov 02 13:29:12.880584 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:12.880561    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-656dd59d89-ml7g4"]
Nov 02 13:29:12.882305 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:29:12.882280    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e12c7b1_dc24_4d4a_9cf4_682f4e82434d.slice/crio-560fce1492c9ef7410b3975a7f3043ef48c9eaf9a4903377beb7b016456caeb9 WatchSource:0}: Error finding container 560fce1492c9ef7410b3975a7f3043ef48c9eaf9a4903377beb7b016456caeb9: Status 404 returned error can't find the container with id 560fce1492c9ef7410b3975a7f3043ef48c9eaf9a4903377beb7b016456caeb9
Nov 02 13:29:13.034702 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:13.034674    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-69b46c5574-d6tqb"]
Nov 02 13:29:13.035768 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:29:13.035739    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd28a3fe_11b1_491e_b7f7_c10276803a4f.slice/crio-567f6fb351fb54923b15ea20e2b1e5f5c3271620073bb84764095f36388306a3 WatchSource:0}: Error finding container 567f6fb351fb54923b15ea20e2b1e5f5c3271620073bb84764095f36388306a3: Status 404 returned error can't find the container with id 567f6fb351fb54923b15ea20e2b1e5f5c3271620073bb84764095f36388306a3
Nov 02 13:29:13.048550 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:13.048527    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"]
Nov 02 13:29:13.051838 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:13.051819    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc4495d7-bhtr2"]
Nov 02 13:29:13.085345 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:13.085316    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-656dd59d89-ml7g4" event={"ID":"2e12c7b1-dc24-4d4a-9cf4-682f4e82434d","Type":"ContainerStarted","Data":"c1b17eeace4be239f3cd6fc71685457901112cdbf3696c50d8d3d41709916dce"}
Nov 02 13:29:13.085468 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:13.085368    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-656dd59d89-ml7g4" event={"ID":"2e12c7b1-dc24-4d4a-9cf4-682f4e82434d","Type":"ContainerStarted","Data":"560fce1492c9ef7410b3975a7f3043ef48c9eaf9a4903377beb7b016456caeb9"}
Nov 02 13:29:13.085468 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:13.085402    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-656dd59d89-ml7g4"
Nov 02 13:29:13.087100 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:13.087078    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-69b46c5574-d6tqb" event={"ID":"cd28a3fe-11b1-491e-b7f7-c10276803a4f","Type":"ContainerStarted","Data":"567f6fb351fb54923b15ea20e2b1e5f5c3271620073bb84764095f36388306a3"}
Nov 02 13:29:13.204883 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:13.204791    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-656dd59d89-ml7g4" podStartSLOduration=1.204778 podStartE2EDuration="1.204778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:29:13.202881144 +0000 UTC m=+5581.766589553" watchObservedRunningTime="2026-11-02 13:29:13.204778 +0000 UTC m=+5581.768486389"
Nov 02 13:29:13.581085 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:13.581050    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a2c581d-6d4d-40a1-9348-12f0e3fb6b38" path="/var/lib/kubelet/pods/8a2c581d-6d4d-40a1-9348-12f0e3fb6b38/volumes"
Nov 02 13:29:14.093470 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:14.093425    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-69b46c5574-d6tqb" event={"ID":"cd28a3fe-11b1-491e-b7f7-c10276803a4f","Type":"ContainerStarted","Data":"61eb48988c3ec8ec0b936d796c98538db1853e3e11a13f253f1c8db2ca88d923"}
Nov 02 13:29:14.093705 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:14.093688    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-69b46c5574-d6tqb"
Nov 02 13:29:14.214020 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:14.213952    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-69b46c5574-d6tqb" podStartSLOduration=2.213938142 podStartE2EDuration="2.213938142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:29:14.21238004 +0000 UTC m=+5582.776088450" watchObservedRunningTime="2026-11-02 13:29:14.213938142 +0000 UTC m=+5582.777646541"
Nov 02 13:29:14.912934 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:14.912905    4464 scope.go:122] "RemoveContainer" containerID="ddaa213feeae439912c593fbf71eab295f4424e906946c71cfc8b4ad349dff64"
Nov 02 13:29:15.185051 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:15.184941    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl" podUID="5c2d3d77-9d5c-457e-b93b-c94cf304e188" containerName="payload-processing" containerID="cri-o://f36c41ef7fc35f2ad29a3f2b386a16d077e9a9f3dd4918191f6ec9eeceef00b6" gracePeriod=25
Nov 02 13:29:15.440191 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:15.440129    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"
Nov 02 13:29:15.561418 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:15.561392    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2d3d77-9d5c-457e-b93b-c94cf304e188-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2d3d77-9d5c-457e-b93b-c94cf304e188-plugins-config-volume\") pod \"5c2d3d77-9d5c-457e-b93b-c94cf304e188\" (UID: \"5c2d3d77-9d5c-457e-b93b-c94cf304e188\") "
Nov 02 13:29:15.561621 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:15.561558    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2d3d77-9d5c-457e-b93b-c94cf304e188-kube-api-access-gztzz\" (UniqueName: \"kubernetes.io/projected/5c2d3d77-9d5c-457e-b93b-c94cf304e188-kube-api-access-gztzz\") pod \"5c2d3d77-9d5c-457e-b93b-c94cf304e188\" (UID: \"5c2d3d77-9d5c-457e-b93b-c94cf304e188\") "
Nov 02 13:29:15.561734 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:15.561710    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2d3d77-9d5c-457e-b93b-c94cf304e188-plugins-config-volume" pod "5c2d3d77-9d5c-457e-b93b-c94cf304e188" (UID: "5c2d3d77-9d5c-457e-b93b-c94cf304e188"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:15.562031 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:15.562007    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2d3d77-9d5c-457e-b93b-c94cf304e188-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:15.564333 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:15.564305    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2d3d77-9d5c-457e-b93b-c94cf304e188-kube-api-access-gztzz" pod "5c2d3d77-9d5c-457e-b93b-c94cf304e188" (UID: "5c2d3d77-9d5c-457e-b93b-c94cf304e188"). InnerVolumeSpecName "kube-api-access-gztzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:15.663477 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:15.663443    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gztzz\" (UniqueName: \"kubernetes.io/projected/5c2d3d77-9d5c-457e-b93b-c94cf304e188-kube-api-access-gztzz\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:16.105488 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:16.105452    4464 generic.go:356] "Generic (PLEG): container finished" podID="5c2d3d77-9d5c-457e-b93b-c94cf304e188" containerID="f36c41ef7fc35f2ad29a3f2b386a16d077e9a9f3dd4918191f6ec9eeceef00b6" exitCode=0
Nov 02 13:29:16.105671 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:16.105503    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"
Nov 02 13:29:16.105671 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:16.105513    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl" event={"ID":"5c2d3d77-9d5c-457e-b93b-c94cf304e188","Type":"ContainerDied","Data":"f36c41ef7fc35f2ad29a3f2b386a16d077e9a9f3dd4918191f6ec9eeceef00b6"}
Nov 02 13:29:16.105671 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:16.105566    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-z7qfl" event={"ID":"5c2d3d77-9d5c-457e-b93b-c94cf304e188","Type":"ContainerDied","Data":"7e8a7f3a0b542ac3d89169bd761a0511189ef38e8a6bed83aed29fbae38832a8"}
Nov 02 13:29:16.105671 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:16.105596    4464 scope.go:122] "RemoveContainer" containerID="f36c41ef7fc35f2ad29a3f2b386a16d077e9a9f3dd4918191f6ec9eeceef00b6"
Nov 02 13:29:16.114088 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:16.114067    4464 scope.go:122] "RemoveContainer" containerID="f36c41ef7fc35f2ad29a3f2b386a16d077e9a9f3dd4918191f6ec9eeceef00b6"
Nov 02 13:29:16.114346 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:29:16.114322    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36c41ef7fc35f2ad29a3f2b386a16d077e9a9f3dd4918191f6ec9eeceef00b6\": container with ID starting with f36c41ef7fc35f2ad29a3f2b386a16d077e9a9f3dd4918191f6ec9eeceef00b6 not found: ID does not exist" containerID="f36c41ef7fc35f2ad29a3f2b386a16d077e9a9f3dd4918191f6ec9eeceef00b6"
Nov 02 13:29:16.114458 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:16.114352    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36c41ef7fc35f2ad29a3f2b386a16d077e9a9f3dd4918191f6ec9eeceef00b6"} err="failed to get container status \"f36c41ef7fc35f2ad29a3f2b386a16d077e9a9f3dd4918191f6ec9eeceef00b6\": rpc error: code = NotFound desc = could not find container \"f36c41ef7fc35f2ad29a3f2b386a16d077e9a9f3dd4918191f6ec9eeceef00b6\": container with ID starting with f36c41ef7fc35f2ad29a3f2b386a16d077e9a9f3dd4918191f6ec9eeceef00b6 not found: ID does not exist"
Nov 02 13:29:16.272765 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:16.272725    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"]
Nov 02 13:29:16.323345 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:16.323314    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-z7qfl"]
Nov 02 13:29:17.580696 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:17.580662    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2d3d77-9d5c-457e-b93b-c94cf304e188" path="/var/lib/kubelet/pods/5c2d3d77-9d5c-457e-b93b-c94cf304e188/volumes"
Nov 02 13:29:24.095989 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:24.095949    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-656dd59d89-ml7g4"
Nov 02 13:29:24.266774 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:24.266740    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5d8fcd46d4-q24jg"]
Nov 02 13:29:25.101598 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:25.101566    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-69b46c5574-d6tqb"
Nov 02 13:29:25.275918 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:25.275883    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-58769f698d-9hnjj"]
Nov 02 13:29:28.290624 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:29:28.290581    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:29:29.267606 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:29.267505    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg" podUID="bde9e4c9-6824-4557-b085-b13066e25ea4" containerName="payload-processing" containerID="cri-o://f4abbf03a4f6885dd312cd34107cb7a92c36b51969ecc9a7685276c18f56c94b" gracePeriod=25
Nov 02 13:29:29.533207 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:29.533144    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg"
Nov 02 13:29:29.595468 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:29.595438    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bde9e4c9-6824-4557-b085-b13066e25ea4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bde9e4c9-6824-4557-b085-b13066e25ea4-plugins-config-volume\") pod \"bde9e4c9-6824-4557-b085-b13066e25ea4\" (UID: \"bde9e4c9-6824-4557-b085-b13066e25ea4\") "
Nov 02 13:29:29.595609 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:29.595481    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde9e4c9-6824-4557-b085-b13066e25ea4-kube-api-access-plpts\" (UniqueName: \"kubernetes.io/projected/bde9e4c9-6824-4557-b085-b13066e25ea4-kube-api-access-plpts\") pod \"bde9e4c9-6824-4557-b085-b13066e25ea4\" (UID: \"bde9e4c9-6824-4557-b085-b13066e25ea4\") "
Nov 02 13:29:29.595796 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:29.595774    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde9e4c9-6824-4557-b085-b13066e25ea4-plugins-config-volume" pod "bde9e4c9-6824-4557-b085-b13066e25ea4" (UID: "bde9e4c9-6824-4557-b085-b13066e25ea4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:29.597640 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:29.597617    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde9e4c9-6824-4557-b085-b13066e25ea4-kube-api-access-plpts" pod "bde9e4c9-6824-4557-b085-b13066e25ea4" (UID: "bde9e4c9-6824-4557-b085-b13066e25ea4"). InnerVolumeSpecName "kube-api-access-plpts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:29.696993 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:29.696945    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bde9e4c9-6824-4557-b085-b13066e25ea4-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:29.697164 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:29.697007    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plpts\" (UniqueName: \"kubernetes.io/projected/bde9e4c9-6824-4557-b085-b13066e25ea4-kube-api-access-plpts\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:30.174559 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:30.174518    4464 generic.go:356] "Generic (PLEG): container finished" podID="bde9e4c9-6824-4557-b085-b13066e25ea4" containerID="f4abbf03a4f6885dd312cd34107cb7a92c36b51969ecc9a7685276c18f56c94b" exitCode=0
Nov 02 13:29:30.174753 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:30.174565    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg"
Nov 02 13:29:30.174753 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:30.174585    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg" event={"ID":"bde9e4c9-6824-4557-b085-b13066e25ea4","Type":"ContainerDied","Data":"f4abbf03a4f6885dd312cd34107cb7a92c36b51969ecc9a7685276c18f56c94b"}
Nov 02 13:29:30.174753 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:30.174645    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5d8fcd46d4-q24jg" event={"ID":"bde9e4c9-6824-4557-b085-b13066e25ea4","Type":"ContainerDied","Data":"1d160233ff1c0fc2e1d792f54c395794f1d3fdb0a0707d4eb63004c1b3ae7b01"}
Nov 02 13:29:30.174753 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:30.174675    4464 scope.go:122] "RemoveContainer" containerID="f4abbf03a4f6885dd312cd34107cb7a92c36b51969ecc9a7685276c18f56c94b"
Nov 02 13:29:30.183805 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:30.183788    4464 scope.go:122] "RemoveContainer" containerID="f4abbf03a4f6885dd312cd34107cb7a92c36b51969ecc9a7685276c18f56c94b"
Nov 02 13:29:30.184065 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:29:30.184048    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4abbf03a4f6885dd312cd34107cb7a92c36b51969ecc9a7685276c18f56c94b\": container with ID starting with f4abbf03a4f6885dd312cd34107cb7a92c36b51969ecc9a7685276c18f56c94b not found: ID does not exist" containerID="f4abbf03a4f6885dd312cd34107cb7a92c36b51969ecc9a7685276c18f56c94b"
Nov 02 13:29:30.184121 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:30.184073    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4abbf03a4f6885dd312cd34107cb7a92c36b51969ecc9a7685276c18f56c94b"} err="failed to get container status \"f4abbf03a4f6885dd312cd34107cb7a92c36b51969ecc9a7685276c18f56c94b\": rpc error: code = NotFound desc = could not find container \"f4abbf03a4f6885dd312cd34107cb7a92c36b51969ecc9a7685276c18f56c94b\": container with ID starting with f4abbf03a4f6885dd312cd34107cb7a92c36b51969ecc9a7685276c18f56c94b not found: ID does not exist"
Nov 02 13:29:30.276725 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:30.276691    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj" podUID="bd7e087d-7840-4930-bc46-e39d15f4b00e" containerName="payload-pre-processing" containerID="cri-o://cf02793df04d2691be6c0a38fccdfeddf4733d64d074b0c346725d6e4f737596" gracePeriod=25
Nov 02 13:29:30.340680 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:30.340648    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5d8fcd46d4-q24jg"]
Nov 02 13:29:30.344944 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:30.344918    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5d8fcd46d4-q24jg"]
Nov 02 13:29:30.516273 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:30.516250    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj"
Nov 02 13:29:30.607043 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:30.607013    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd7e087d-7840-4930-bc46-e39d15f4b00e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7e087d-7840-4930-bc46-e39d15f4b00e-plugins-config-volume\") pod \"bd7e087d-7840-4930-bc46-e39d15f4b00e\" (UID: \"bd7e087d-7840-4930-bc46-e39d15f4b00e\") "
Nov 02 13:29:30.607515 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:30.607071    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7e087d-7840-4930-bc46-e39d15f4b00e-kube-api-access-bwktf\" (UniqueName: \"kubernetes.io/projected/bd7e087d-7840-4930-bc46-e39d15f4b00e-kube-api-access-bwktf\") pod \"bd7e087d-7840-4930-bc46-e39d15f4b00e\" (UID: \"bd7e087d-7840-4930-bc46-e39d15f4b00e\") "
Nov 02 13:29:30.607515 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:30.607449    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7e087d-7840-4930-bc46-e39d15f4b00e-plugins-config-volume" pod "bd7e087d-7840-4930-bc46-e39d15f4b00e" (UID: "bd7e087d-7840-4930-bc46-e39d15f4b00e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:30.609052 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:30.609031    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7e087d-7840-4930-bc46-e39d15f4b00e-kube-api-access-bwktf" pod "bd7e087d-7840-4930-bc46-e39d15f4b00e" (UID: "bd7e087d-7840-4930-bc46-e39d15f4b00e"). InnerVolumeSpecName "kube-api-access-bwktf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:30.708381 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:30.708296    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwktf\" (UniqueName: \"kubernetes.io/projected/bd7e087d-7840-4930-bc46-e39d15f4b00e-kube-api-access-bwktf\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:30.708381 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:30.708325    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7e087d-7840-4930-bc46-e39d15f4b00e-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:31.181279 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:31.181244    4464 generic.go:356] "Generic (PLEG): container finished" podID="bd7e087d-7840-4930-bc46-e39d15f4b00e" containerID="cf02793df04d2691be6c0a38fccdfeddf4733d64d074b0c346725d6e4f737596" exitCode=0
Nov 02 13:29:31.181473 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:31.181301    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj"
Nov 02 13:29:31.181473 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:31.181333    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj" event={"ID":"bd7e087d-7840-4930-bc46-e39d15f4b00e","Type":"ContainerDied","Data":"cf02793df04d2691be6c0a38fccdfeddf4733d64d074b0c346725d6e4f737596"}
Nov 02 13:29:31.181473 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:31.181413    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-58769f698d-9hnjj" event={"ID":"bd7e087d-7840-4930-bc46-e39d15f4b00e","Type":"ContainerDied","Data":"5b99f663721e294aa8246e5d96e7ef4d6ec4e2ba1f16f53b8f3b4eccc2644d63"}
Nov 02 13:29:31.181473 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:31.181437    4464 scope.go:122] "RemoveContainer" containerID="cf02793df04d2691be6c0a38fccdfeddf4733d64d074b0c346725d6e4f737596"
Nov 02 13:29:31.190409 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:31.190391    4464 scope.go:122] "RemoveContainer" containerID="cf02793df04d2691be6c0a38fccdfeddf4733d64d074b0c346725d6e4f737596"
Nov 02 13:29:31.190652 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:29:31.190633    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf02793df04d2691be6c0a38fccdfeddf4733d64d074b0c346725d6e4f737596\": container with ID starting with cf02793df04d2691be6c0a38fccdfeddf4733d64d074b0c346725d6e4f737596 not found: ID does not exist" containerID="cf02793df04d2691be6c0a38fccdfeddf4733d64d074b0c346725d6e4f737596"
Nov 02 13:29:31.190706 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:31.190658    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf02793df04d2691be6c0a38fccdfeddf4733d64d074b0c346725d6e4f737596"} err="failed to get container status \"cf02793df04d2691be6c0a38fccdfeddf4733d64d074b0c346725d6e4f737596\": rpc error: code = NotFound desc = could not find container \"cf02793df04d2691be6c0a38fccdfeddf4733d64d074b0c346725d6e4f737596\": container with ID starting with cf02793df04d2691be6c0a38fccdfeddf4733d64d074b0c346725d6e4f737596 not found: ID does not exist"
Nov 02 13:29:31.355672 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:31.355645    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-58769f698d-9hnjj"]
Nov 02 13:29:31.363352 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:31.363329    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-58769f698d-9hnjj"]
Nov 02 13:29:31.582867 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:31.582837    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd7e087d-7840-4930-bc46-e39d15f4b00e" path="/var/lib/kubelet/pods/bd7e087d-7840-4930-bc46-e39d15f4b00e/volumes"
Nov 02 13:29:31.583255 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:31.583241    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bde9e4c9-6824-4557-b085-b13066e25ea4" path="/var/lib/kubelet/pods/bde9e4c9-6824-4557-b085-b13066e25ea4/volumes"
Nov 02 13:29:33.343787 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.343759    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"]
Nov 02 13:29:33.344271 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.344255    4464 cpu_manager.go:414] "containerMap: removing container" podUID="5c2d3d77-9d5c-457e-b93b-c94cf304e188" containerName="payload-processing"
Nov 02 13:29:33.344323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.344273    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2d3d77-9d5c-457e-b93b-c94cf304e188" containerName="payload-processing"
Nov 02 13:29:33.344323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.344289    4464 cpu_manager.go:414] "containerMap: removing container" podUID="bde9e4c9-6824-4557-b085-b13066e25ea4" containerName="payload-processing"
Nov 02 13:29:33.344323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.344297    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde9e4c9-6824-4557-b085-b13066e25ea4" containerName="payload-processing"
Nov 02 13:29:33.344323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.344308    4464 cpu_manager.go:414] "containerMap: removing container" podUID="bd7e087d-7840-4930-bc46-e39d15f4b00e" containerName="payload-pre-processing"
Nov 02 13:29:33.344323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.344313    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd7e087d-7840-4930-bc46-e39d15f4b00e" containerName="payload-pre-processing"
Nov 02 13:29:33.344484 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.344410    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2d3d77-9d5c-457e-b93b-c94cf304e188" containerName="payload-processing"
Nov 02 13:29:33.344484 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.344421    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde9e4c9-6824-4557-b085-b13066e25ea4" containerName="payload-processing"
Nov 02 13:29:33.344484 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.344428    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd7e087d-7840-4930-bc46-e39d15f4b00e" containerName="payload-pre-processing"
Nov 02 13:29:33.347357 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.347341    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:29:33.403929 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.403903    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 13:29:33.404131 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.403998    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\""
Nov 02 13:29:33.433003 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.432962    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-x2xm2\" (UID: \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:29:33.433168 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.433030    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs7w9\" (UniqueName: \"kubernetes.io/projected/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-kube-api-access-hs7w9\") pod \"payload-processing-external-model-5bcd7997-x2xm2\" (UID: \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:29:33.433168 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.433157    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-x2xm2\" (UID: \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:29:33.433299 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.433205    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-x2xm2\" (UID: \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:29:33.454517 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.454489    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 13:29:33.454636 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.454599    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 13:29:33.454636 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.454605    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 13:29:33.454764 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.454664    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-k9ngm\""
Nov 02 13:29:33.456570 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.456551    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"]
Nov 02 13:29:33.534272 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.534249    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-x2xm2\" (UID: \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:29:33.534380 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.534295    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs7w9\" (UniqueName: \"kubernetes.io/projected/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-kube-api-access-hs7w9\") pod \"payload-processing-external-model-5bcd7997-x2xm2\" (UID: \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:29:33.534380 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.534349    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-x2xm2\" (UID: \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:29:33.534456 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.534382    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-x2xm2\" (UID: \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:29:33.535044 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.535026    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-x2xm2\" (UID: \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:29:33.535141 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.535106    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-x2xm2\" (UID: \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:29:33.536703 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.536683    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-x2xm2\" (UID: \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:29:33.599248 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.599182    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs7w9\" (UniqueName: \"kubernetes.io/projected/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-kube-api-access-hs7w9\") pod \"payload-processing-external-model-5bcd7997-x2xm2\" (UID: \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:29:33.660129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.660105    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:29:33.910971 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:33.910892    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:29:34.031992 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:34.031947    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"]
Nov 02 13:29:34.033969 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:29:34.033941    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17690f95_d8a7_4362_80ef_ddc3ae7ccca5.slice/crio-9187844d227a5e39f63136aa8b8b74e6e63672aef4a47c42ab7b26b73c309c19 WatchSource:0}: Error finding container 9187844d227a5e39f63136aa8b8b74e6e63672aef4a47c42ab7b26b73c309c19: Status 404 returned error can't find the container with id 9187844d227a5e39f63136aa8b8b74e6e63672aef4a47c42ab7b26b73c309c19
Nov 02 13:29:34.201580 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:34.201497    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2" event={"ID":"17690f95-d8a7-4362-80ef-ddc3ae7ccca5","Type":"ContainerStarted","Data":"5734a52213e68d506e5e1ec1e2820d6af03c903a5578d17d07d98aa04ab5ec9e"}
Nov 02 13:29:34.201580 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:34.201552    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2" event={"ID":"17690f95-d8a7-4362-80ef-ddc3ae7ccca5","Type":"ContainerStarted","Data":"9187844d227a5e39f63136aa8b8b74e6e63672aef4a47c42ab7b26b73c309c19"}
Nov 02 13:29:34.201769 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:34.201637    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:29:34.321612 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:34.321559    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2" podStartSLOduration=1.321542435 podStartE2EDuration="1.321542435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:29:34.319325449 +0000 UTC m=+5602.883033858" watchObservedRunningTime="2026-11-02 13:29:34.321542435 +0000 UTC m=+5602.885250861"
Nov 02 13:29:37.374999 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:37.374052    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ppqp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:29:37.374999 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:37.374118    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4" podUID="e5780a48-9777-4e56-87b4-e0d00bebb615" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:29:37.677054 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:37.675088    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7xbnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:29:37.677054 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:37.675169    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw" podUID="931c4f3d-1712-4c42-9226-ec630f19f7ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:29:45.211417 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:45.211390    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:29:47.248457 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:47.248422    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"]
Nov 02 13:29:47.248916 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:47.248615    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4" podUID="e5780a48-9777-4e56-87b4-e0d00bebb615" containerName="istio-proxy" containerID="cri-o://f636c4ad6cb07893d2e3492b3a3d49d16dc6e026a5f8f80d837b629e6f4f2ec8" gracePeriod=30
Nov 02 13:29:47.298944 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:47.298908    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"]
Nov 02 13:29:47.299128 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:47.299108    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw" podUID="931c4f3d-1712-4c42-9226-ec630f19f7ca" containerName="istio-proxy" containerID="cri-o://cbf6f5e093692f19a40aa6688bb1759ccc1429d5259ece4df8c79a6211818aa2" gracePeriod=30
Nov 02 13:29:51.325527 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:51.325490    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ppqp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 02 13:29:51.325921 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:51.325560    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4" podUID="e5780a48-9777-4e56-87b4-e0d00bebb615" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:29:51.625914 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:51.625882    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-7xbnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 02 13:29:51.626108 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:51.625930    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw" podUID="931c4f3d-1712-4c42-9226-ec630f19f7ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:29:52.545760 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.545699    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:29:52.555481 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.555452    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-podinfo\") pod \"e5780a48-9777-4e56-87b4-e0d00bebb615\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") "
Nov 02 13:29:52.555572 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.555538    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-credential-socket\") pod \"e5780a48-9777-4e56-87b4-e0d00bebb615\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") "
Nov 02 13:29:52.555612 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.555587    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-data\") pod \"e5780a48-9777-4e56-87b4-e0d00bebb615\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") "
Nov 02 13:29:52.555664 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.555645    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-token\" (UniqueName: \"kubernetes.io/projected/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-token\") pod \"e5780a48-9777-4e56-87b4-e0d00bebb615\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") "
Nov 02 13:29:52.555724 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.555680    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5780a48-9777-4e56-87b4-e0d00bebb615-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5780a48-9777-4e56-87b4-e0d00bebb615-istiod-ca-cert\") pod \"e5780a48-9777-4e56-87b4-e0d00bebb615\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") "
Nov 02 13:29:52.555724 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.555714    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-envoy\") pod \"e5780a48-9777-4e56-87b4-e0d00bebb615\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") "
Nov 02 13:29:52.555845 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.555756    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-workload-socket\") pod \"e5780a48-9777-4e56-87b4-e0d00bebb615\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") "
Nov 02 13:29:52.555905 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.555817    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-credential-socket" pod "e5780a48-9777-4e56-87b4-e0d00bebb615" (UID: "e5780a48-9777-4e56-87b4-e0d00bebb615"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:52.555905 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.555872    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-workload-certs\") pod \"e5780a48-9777-4e56-87b4-e0d00bebb615\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") "
Nov 02 13:29:52.556060 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.555918    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5780a48-9777-4e56-87b4-e0d00bebb615-kube-api-access-4zsz5\" (UniqueName: \"kubernetes.io/projected/e5780a48-9777-4e56-87b4-e0d00bebb615-kube-api-access-4zsz5\") pod \"e5780a48-9777-4e56-87b4-e0d00bebb615\" (UID: \"e5780a48-9777-4e56-87b4-e0d00bebb615\") "
Nov 02 13:29:52.556060 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.555904    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-data" pod "e5780a48-9777-4e56-87b4-e0d00bebb615" (UID: "e5780a48-9777-4e56-87b4-e0d00bebb615"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:52.556060 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.556036    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-workload-socket" pod "e5780a48-9777-4e56-87b4-e0d00bebb615" (UID: "e5780a48-9777-4e56-87b4-e0d00bebb615"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:52.556238 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.556143    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-workload-certs" pod "e5780a48-9777-4e56-87b4-e0d00bebb615" (UID: "e5780a48-9777-4e56-87b4-e0d00bebb615"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:52.556238 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.556175    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5780a48-9777-4e56-87b4-e0d00bebb615-istiod-ca-cert" pod "e5780a48-9777-4e56-87b4-e0d00bebb615" (UID: "e5780a48-9777-4e56-87b4-e0d00bebb615"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:52.556364 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.556343    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5780a48-9777-4e56-87b4-e0d00bebb615-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:52.556427 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.556365    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:52.556427 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.556379    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:52.556427 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.556393    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:52.556427 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.556406    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:52.557893 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.557865    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-token" pod "e5780a48-9777-4e56-87b4-e0d00bebb615" (UID: "e5780a48-9777-4e56-87b4-e0d00bebb615"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:52.558354 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.558332    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-podinfo" pod "e5780a48-9777-4e56-87b4-e0d00bebb615" (UID: "e5780a48-9777-4e56-87b4-e0d00bebb615"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:29:52.558451 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.558431    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5780a48-9777-4e56-87b4-e0d00bebb615-kube-api-access-4zsz5" pod "e5780a48-9777-4e56-87b4-e0d00bebb615" (UID: "e5780a48-9777-4e56-87b4-e0d00bebb615"). InnerVolumeSpecName "kube-api-access-4zsz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:52.559224 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.559200    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-envoy" pod "e5780a48-9777-4e56-87b4-e0d00bebb615" (UID: "e5780a48-9777-4e56-87b4-e0d00bebb615"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:52.567683 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.567666    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:29:52.657489 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.657463    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-data\") pod \"931c4f3d-1712-4c42-9226-ec630f19f7ca\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") "
Nov 02 13:29:52.657610 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.657499    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-envoy\") pod \"931c4f3d-1712-4c42-9226-ec630f19f7ca\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") "
Nov 02 13:29:52.657610 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.657559    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/931c4f3d-1712-4c42-9226-ec630f19f7ca-kube-api-access-nqr7m\" (UniqueName: \"kubernetes.io/projected/931c4f3d-1712-4c42-9226-ec630f19f7ca-kube-api-access-nqr7m\") pod \"931c4f3d-1712-4c42-9226-ec630f19f7ca\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") "
Nov 02 13:29:52.657734 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.657626    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-workload-socket\") pod \"931c4f3d-1712-4c42-9226-ec630f19f7ca\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") "
Nov 02 13:29:52.657734 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.657680    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/931c4f3d-1712-4c42-9226-ec630f19f7ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931c4f3d-1712-4c42-9226-ec630f19f7ca-istiod-ca-cert\") pod \"931c4f3d-1712-4c42-9226-ec630f19f7ca\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") "
Nov 02 13:29:52.657734 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.657706    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-data" pod "931c4f3d-1712-4c42-9226-ec630f19f7ca" (UID: "931c4f3d-1712-4c42-9226-ec630f19f7ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:52.657897 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.657737    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-token\" (UniqueName: \"kubernetes.io/projected/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-token\") pod \"931c4f3d-1712-4c42-9226-ec630f19f7ca\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") "
Nov 02 13:29:52.657897 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.657772    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-credential-socket\") pod \"931c4f3d-1712-4c42-9226-ec630f19f7ca\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") "
Nov 02 13:29:52.657897 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.657824    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-workload-certs\") pod \"931c4f3d-1712-4c42-9226-ec630f19f7ca\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") "
Nov 02 13:29:52.657897 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.657862    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-podinfo\") pod \"931c4f3d-1712-4c42-9226-ec630f19f7ca\" (UID: \"931c4f3d-1712-4c42-9226-ec630f19f7ca\") "
Nov 02 13:29:52.658147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.657916    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-workload-socket" pod "931c4f3d-1712-4c42-9226-ec630f19f7ca" (UID: "931c4f3d-1712-4c42-9226-ec630f19f7ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:52.658147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.658048    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-credential-socket" pod "931c4f3d-1712-4c42-9226-ec630f19f7ca" (UID: "931c4f3d-1712-4c42-9226-ec630f19f7ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:52.658147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.658080    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931c4f3d-1712-4c42-9226-ec630f19f7ca-istiod-ca-cert" pod "931c4f3d-1712-4c42-9226-ec630f19f7ca" (UID: "931c4f3d-1712-4c42-9226-ec630f19f7ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:52.658322 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.658157    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-workload-certs" pod "931c4f3d-1712-4c42-9226-ec630f19f7ca" (UID: "931c4f3d-1712-4c42-9226-ec630f19f7ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:52.658322 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.658283    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:52.658322 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.658296    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:52.658322 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.658306    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:52.658322 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.658320    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931c4f3d-1712-4c42-9226-ec630f19f7ca-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:52.658549 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.658333    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:52.658549 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.658342    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zsz5\" (UniqueName: \"kubernetes.io/projected/e5780a48-9777-4e56-87b4-e0d00bebb615-kube-api-access-4zsz5\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:52.658549 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.658357    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:52.658549 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.658365    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5780a48-9777-4e56-87b4-e0d00bebb615-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:52.658549 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.658373    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:52.659734 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.659711    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-token" pod "931c4f3d-1712-4c42-9226-ec630f19f7ca" (UID: "931c4f3d-1712-4c42-9226-ec630f19f7ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:52.659989 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.659946    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-podinfo" pod "931c4f3d-1712-4c42-9226-ec630f19f7ca" (UID: "931c4f3d-1712-4c42-9226-ec630f19f7ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:29:52.660161 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.660144    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931c4f3d-1712-4c42-9226-ec630f19f7ca-kube-api-access-nqr7m" pod "931c4f3d-1712-4c42-9226-ec630f19f7ca" (UID: "931c4f3d-1712-4c42-9226-ec630f19f7ca"). InnerVolumeSpecName "kube-api-access-nqr7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:52.660203 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.660161    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-envoy" pod "931c4f3d-1712-4c42-9226-ec630f19f7ca" (UID: "931c4f3d-1712-4c42-9226-ec630f19f7ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:29:52.759034 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.759009    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:52.759034 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.759032    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:52.759190 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.759042    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqr7m\" (UniqueName: \"kubernetes.io/projected/931c4f3d-1712-4c42-9226-ec630f19f7ca-kube-api-access-nqr7m\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:52.759190 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:52.759051    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931c4f3d-1712-4c42-9226-ec630f19f7ca-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:29:53.318325 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.318290    4464 generic.go:356] "Generic (PLEG): container finished" podID="931c4f3d-1712-4c42-9226-ec630f19f7ca" containerID="cbf6f5e093692f19a40aa6688bb1759ccc1429d5259ece4df8c79a6211818aa2" exitCode=0
Nov 02 13:29:53.318481 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.318339    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"
Nov 02 13:29:53.318481 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.318380    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw" event={"ID":"931c4f3d-1712-4c42-9226-ec630f19f7ca","Type":"ContainerDied","Data":"cbf6f5e093692f19a40aa6688bb1759ccc1429d5259ece4df8c79a6211818aa2"}
Nov 02 13:29:53.318481 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.318436    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw" event={"ID":"931c4f3d-1712-4c42-9226-ec630f19f7ca","Type":"ContainerDied","Data":"9a2b0bc04239fe0a9b08b0231d6a182cbf1b1fa5640cac1dd2cd34d6128ed97a"}
Nov 02 13:29:53.318481 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.318467    4464 scope.go:122] "RemoveContainer" containerID="cbf6f5e093692f19a40aa6688bb1759ccc1429d5259ece4df8c79a6211818aa2"
Nov 02 13:29:53.320031 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.320010    4464 generic.go:356] "Generic (PLEG): container finished" podID="e5780a48-9777-4e56-87b4-e0d00bebb615" containerID="f636c4ad6cb07893d2e3492b3a3d49d16dc6e026a5f8f80d837b629e6f4f2ec8" exitCode=0
Nov 02 13:29:53.320133 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.320048    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4" event={"ID":"e5780a48-9777-4e56-87b4-e0d00bebb615","Type":"ContainerDied","Data":"f636c4ad6cb07893d2e3492b3a3d49d16dc6e026a5f8f80d837b629e6f4f2ec8"}
Nov 02 13:29:53.320133 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.320094    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4" event={"ID":"e5780a48-9777-4e56-87b4-e0d00bebb615","Type":"ContainerDied","Data":"abff9ec801027dbfdd460e7b853dc4e522936e319c0edb14eb555c68c72975bd"}
Nov 02 13:29:53.320133 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.320097    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"
Nov 02 13:29:53.328773 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.328757    4464 scope.go:122] "RemoveContainer" containerID="cbf6f5e093692f19a40aa6688bb1759ccc1429d5259ece4df8c79a6211818aa2"
Nov 02 13:29:53.329183 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:29:53.329162    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf6f5e093692f19a40aa6688bb1759ccc1429d5259ece4df8c79a6211818aa2\": container with ID starting with cbf6f5e093692f19a40aa6688bb1759ccc1429d5259ece4df8c79a6211818aa2 not found: ID does not exist" containerID="cbf6f5e093692f19a40aa6688bb1759ccc1429d5259ece4df8c79a6211818aa2"
Nov 02 13:29:53.329307 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.329193    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf6f5e093692f19a40aa6688bb1759ccc1429d5259ece4df8c79a6211818aa2"} err="failed to get container status \"cbf6f5e093692f19a40aa6688bb1759ccc1429d5259ece4df8c79a6211818aa2\": rpc error: code = NotFound desc = could not find container \"cbf6f5e093692f19a40aa6688bb1759ccc1429d5259ece4df8c79a6211818aa2\": container with ID starting with cbf6f5e093692f19a40aa6688bb1759ccc1429d5259ece4df8c79a6211818aa2 not found: ID does not exist"
Nov 02 13:29:53.329307 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.329214    4464 scope.go:122] "RemoveContainer" containerID="f636c4ad6cb07893d2e3492b3a3d49d16dc6e026a5f8f80d837b629e6f4f2ec8"
Nov 02 13:29:53.339298 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.339278    4464 scope.go:122] "RemoveContainer" containerID="f636c4ad6cb07893d2e3492b3a3d49d16dc6e026a5f8f80d837b629e6f4f2ec8"
Nov 02 13:29:53.339578 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:29:53.339559    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f636c4ad6cb07893d2e3492b3a3d49d16dc6e026a5f8f80d837b629e6f4f2ec8\": container with ID starting with f636c4ad6cb07893d2e3492b3a3d49d16dc6e026a5f8f80d837b629e6f4f2ec8 not found: ID does not exist" containerID="f636c4ad6cb07893d2e3492b3a3d49d16dc6e026a5f8f80d837b629e6f4f2ec8"
Nov 02 13:29:53.339640 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.339584    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f636c4ad6cb07893d2e3492b3a3d49d16dc6e026a5f8f80d837b629e6f4f2ec8"} err="failed to get container status \"f636c4ad6cb07893d2e3492b3a3d49d16dc6e026a5f8f80d837b629e6f4f2ec8\": rpc error: code = NotFound desc = could not find container \"f636c4ad6cb07893d2e3492b3a3d49d16dc6e026a5f8f80d837b629e6f4f2ec8\": container with ID starting with f636c4ad6cb07893d2e3492b3a3d49d16dc6e026a5f8f80d837b629e6f4f2ec8 not found: ID does not exist"
Nov 02 13:29:53.487395 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.487364    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"]
Nov 02 13:29:53.538113 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.538087    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-7xbnw"]
Nov 02 13:29:53.582674 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.582607    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="931c4f3d-1712-4c42-9226-ec630f19f7ca" path="/var/lib/kubelet/pods/931c4f3d-1712-4c42-9226-ec630f19f7ca/volumes"
Nov 02 13:29:53.603900 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.603879    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"]
Nov 02 13:29:53.609135 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:53.609114    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ppqp4"]
Nov 02 13:29:55.581311 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:29:55.581238    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5780a48-9777-4e56-87b4-e0d00bebb615" path="/var/lib/kubelet/pods/e5780a48-9777-4e56-87b4-e0d00bebb615/volumes"
Nov 02 13:29:58.292044 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:29:58.292007    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:30:00.181390 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.181349    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2"]
Nov 02 13:30:00.181802 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.181769    4464 cpu_manager.go:414] "containerMap: removing container" podUID="931c4f3d-1712-4c42-9226-ec630f19f7ca" containerName="istio-proxy"
Nov 02 13:30:00.181802 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.181780    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="931c4f3d-1712-4c42-9226-ec630f19f7ca" containerName="istio-proxy"
Nov 02 13:30:00.181882 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.181818    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e5780a48-9777-4e56-87b4-e0d00bebb615" containerName="istio-proxy"
Nov 02 13:30:00.181882 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.181823    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5780a48-9777-4e56-87b4-e0d00bebb615" containerName="istio-proxy"
Nov 02 13:30:00.181882 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.181878    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5780a48-9777-4e56-87b4-e0d00bebb615" containerName="istio-proxy"
Nov 02 13:30:00.182007 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.181888    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="931c4f3d-1712-4c42-9226-ec630f19f7ca" containerName="istio-proxy"
Nov 02 13:30:00.184853 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.184838    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2"
Nov 02 13:30:00.222742 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.222711    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bba49fc-8f4f-4f61-9835-c4f51d9f5d09-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893770-kc2k2\" (UID: \"6bba49fc-8f4f-4f61-9835-c4f51d9f5d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2"
Nov 02 13:30:00.222876 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.222854    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwsk7\" (UniqueName: \"kubernetes.io/projected/6bba49fc-8f4f-4f61-9835-c4f51d9f5d09-kube-api-access-dwsk7\") pod \"maas-api-key-cleanup-29893770-kc2k2\" (UID: \"6bba49fc-8f4f-4f61-9835-c4f51d9f5d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2"
Nov 02 13:30:00.240456 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.240433    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:30:00.240597 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.240508    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:30:00.291284 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.291253    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:30:00.293065 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.293042    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2"]
Nov 02 13:30:00.323731 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.323698    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwsk7\" (UniqueName: \"kubernetes.io/projected/6bba49fc-8f4f-4f61-9835-c4f51d9f5d09-kube-api-access-dwsk7\") pod \"maas-api-key-cleanup-29893770-kc2k2\" (UID: \"6bba49fc-8f4f-4f61-9835-c4f51d9f5d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2"
Nov 02 13:30:00.323949 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.323752    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bba49fc-8f4f-4f61-9835-c4f51d9f5d09-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893770-kc2k2\" (UID: \"6bba49fc-8f4f-4f61-9835-c4f51d9f5d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2"
Nov 02 13:30:00.324469 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.324441    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bba49fc-8f4f-4f61-9835-c4f51d9f5d09-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893770-kc2k2\" (UID: \"6bba49fc-8f4f-4f61-9835-c4f51d9f5d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2"
Nov 02 13:30:00.386324 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.386296    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwsk7\" (UniqueName: \"kubernetes.io/projected/6bba49fc-8f4f-4f61-9835-c4f51d9f5d09-kube-api-access-dwsk7\") pod \"maas-api-key-cleanup-29893770-kc2k2\" (UID: \"6bba49fc-8f4f-4f61-9835-c4f51d9f5d09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2"
Nov 02 13:30:00.503605 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.503577    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2"
Nov 02 13:30:00.872491 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:00.872466    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2"]
Nov 02 13:30:00.873882 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:30:00.873852    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bba49fc_8f4f_4f61_9835_c4f51d9f5d09.slice/crio-33eefb95e63ab5517b69f8a73cb3b0a732ebe59d4445f36e8f5e8768991bd90f WatchSource:0}: Error finding container 33eefb95e63ab5517b69f8a73cb3b0a732ebe59d4445f36e8f5e8768991bd90f: Status 404 returned error can't find the container with id 33eefb95e63ab5517b69f8a73cb3b0a732ebe59d4445f36e8f5e8768991bd90f
Nov 02 13:30:01.363493 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:01.363457    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2" event={"ID":"6bba49fc-8f4f-4f61-9835-c4f51d9f5d09","Type":"ContainerStarted","Data":"5fd601800da5d6c891df9d90779beb397a775f1419d4cbcc3a2d7508ac59e702"}
Nov 02 13:30:01.363997 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:01.363509    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2" event={"ID":"6bba49fc-8f4f-4f61-9835-c4f51d9f5d09","Type":"ContainerStarted","Data":"33eefb95e63ab5517b69f8a73cb3b0a732ebe59d4445f36e8f5e8768991bd90f"}
Nov 02 13:30:01.481473 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:01.481420    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2" podStartSLOduration=1.481404994 podStartE2EDuration="1.481404994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:01.480865346 +0000 UTC m=+5630.044573737" watchObservedRunningTime="2026-11-02 13:30:01.481404994 +0000 UTC m=+5630.045113402"
Nov 02 13:30:02.258325 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.258290    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"]
Nov 02 13:30:02.261959 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.261942    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.309121 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.309093    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"]
Nov 02 13:30:02.313096 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.313078    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.341350 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341328    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.341446 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341361    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.341446 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341427    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.341523 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341493    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c134c2-e2f8-4a16-a557-65786f86217c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.341569 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341533    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.341569 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341553    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.341631 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341576    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.341631 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341594    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c134c2-e2f8-4a16-a557-65786f86217c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.341631 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341609    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.341729 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341697    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.341729 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341714    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.341796 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341746    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv8tl\" (UniqueName: \"kubernetes.io/projected/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-kube-api-access-kv8tl\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.341796 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341768    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.341868 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341836    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c134c2-e2f8-4a16-a557-65786f86217c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.341868 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341853    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.341934 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341870    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhwp5\" (UniqueName: \"kubernetes.io/projected/02c134c2-e2f8-4a16-a557-65786f86217c-kube-api-access-zhwp5\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.341934 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341889    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.341934 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.341905    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.366674 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.366646    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"]
Nov 02 13:30:02.370650 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.370628    4464 generic.go:356] "Generic (PLEG): container finished" podID="6bba49fc-8f4f-4f61-9835-c4f51d9f5d09" containerID="5fd601800da5d6c891df9d90779beb397a775f1419d4cbcc3a2d7508ac59e702" exitCode=60
Nov 02 13:30:02.370775 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.370713    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2" event={"ID":"6bba49fc-8f4f-4f61-9835-c4f51d9f5d09","Type":"ContainerDied","Data":"5fd601800da5d6c891df9d90779beb397a775f1419d4cbcc3a2d7508ac59e702"}
Nov 02 13:30:02.370929 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.370912    4464 scope.go:122] "RemoveContainer" containerID="5fd601800da5d6c891df9d90779beb397a775f1419d4cbcc3a2d7508ac59e702"
Nov 02 13:30:02.418439 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.418413    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"]
Nov 02 13:30:02.442433 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.442410    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.442559 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.442446    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.442559 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.442463    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.442815 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.442660    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c134c2-e2f8-4a16-a557-65786f86217c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.442898 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.442817    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.442898 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.442843    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.442898 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.442894    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.443201 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.442896    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.443201 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.442906    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.443201 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.442919    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.443201 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.443175    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.443536 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.443268    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.443536 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.443369    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv8tl\" (UniqueName: \"kubernetes.io/projected/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-kube-api-access-kv8tl\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.443536 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.443411    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.443536 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.443448    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c134c2-e2f8-4a16-a557-65786f86217c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.443787 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.443548    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.443787 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.443593    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhwp5\" (UniqueName: \"kubernetes.io/projected/02c134c2-e2f8-4a16-a557-65786f86217c-kube-api-access-zhwp5\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.443787 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.443648    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.443787 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.443676    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.444041 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.443802    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.444041 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.443839    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.444041 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.443910    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.444041 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.444016    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c134c2-e2f8-4a16-a557-65786f86217c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.444614 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.444273    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.444614 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.444352    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.444755 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.444655    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.445101 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.445056    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.445220 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.445116    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c134c2-e2f8-4a16-a557-65786f86217c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.446247 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.446203    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.446556 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.446524    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c134c2-e2f8-4a16-a557-65786f86217c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.447158 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.447131    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.447482 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.447461    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.505682 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.505653    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhwp5\" (UniqueName: \"kubernetes.io/projected/02c134c2-e2f8-4a16-a557-65786f86217c-kube-api-access-zhwp5\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.505846 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.505763    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv8tl\" (UniqueName: \"kubernetes.io/projected/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-kube-api-access-kv8tl\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.505846 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.505787    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-zfkls\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.508552 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.508482    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c134c2-e2f8-4a16-a557-65786f86217c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-bp4h7\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.574337 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.574305    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:02.625645 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.625618    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:02.997990 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:02.997948    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"]
Nov 02 13:30:03.033739 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:30:03.033711    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b4bfae_a5c2_4eec_ae7a_d3fd55c1f921.slice/crio-f0382797fb81b7d654bccc109f826459e4c2d2e54843f0533f1979a9463219b7 WatchSource:0}: Error finding container f0382797fb81b7d654bccc109f826459e4c2d2e54843f0533f1979a9463219b7: Status 404 returned error can't find the container with id f0382797fb81b7d654bccc109f826459e4c2d2e54843f0533f1979a9463219b7
Nov 02 13:30:03.035915 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.035887    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:30:03.035968 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.035947    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:30:03.036034 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.036013    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:30:03.050779 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.050744    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"]
Nov 02 13:30:03.050779 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:30:03.050758    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c134c2_e2f8_4a16_a557_65786f86217c.slice/crio-20183dbb830d4420c77d163349aeff31c0c1f742d3fae797a81e499a11612457 WatchSource:0}: Error finding container 20183dbb830d4420c77d163349aeff31c0c1f742d3fae797a81e499a11612457: Status 404 returned error can't find the container with id 20183dbb830d4420c77d163349aeff31c0c1f742d3fae797a81e499a11612457
Nov 02 13:30:03.052932 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.052903    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:30:03.053077 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.052995    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:30:03.053156 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.053135    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:30:03.379840 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.379806    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7" event={"ID":"02c134c2-e2f8-4a16-a557-65786f86217c","Type":"ContainerStarted","Data":"7a94b036398001bc4545922a1913a86dbc8ffecd61a4a6bb5be77b6db2f448fc"}
Nov 02 13:30:03.380438 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.380407    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7" event={"ID":"02c134c2-e2f8-4a16-a557-65786f86217c","Type":"ContainerStarted","Data":"20183dbb830d4420c77d163349aeff31c0c1f742d3fae797a81e499a11612457"}
Nov 02 13:30:03.382217 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.382164    4464 generic.go:356] "Generic (PLEG): container finished" podID="6bba49fc-8f4f-4f61-9835-c4f51d9f5d09" containerID="3c1f721f8ef108284e8f9678daa70bb939ad9f3be9f7b8fe77052cb260b45f95" exitCode=60
Nov 02 13:30:03.382351 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.382208    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2" event={"ID":"6bba49fc-8f4f-4f61-9835-c4f51d9f5d09","Type":"ContainerDied","Data":"3c1f721f8ef108284e8f9678daa70bb939ad9f3be9f7b8fe77052cb260b45f95"}
Nov 02 13:30:03.382351 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.382268    4464 scope.go:122] "RemoveContainer" containerID="5fd601800da5d6c891df9d90779beb397a775f1419d4cbcc3a2d7508ac59e702"
Nov 02 13:30:03.382479 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.382460    4464 scope.go:122] "RemoveContainer" containerID="3c1f721f8ef108284e8f9678daa70bb939ad9f3be9f7b8fe77052cb260b45f95"
Nov 02 13:30:03.382701 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:30:03.382681    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893770-kc2k2_odh-ai-gateway-infra(6bba49fc-8f4f-4f61-9835-c4f51d9f5d09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2" podUID="6bba49fc-8f4f-4f61-9835-c4f51d9f5d09"
Nov 02 13:30:03.384989 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.384937    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls" event={"ID":"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921","Type":"ContainerStarted","Data":"b2afcfa3c234579a7156db6bffb43032ffdfdbf6a25513bfc48e923ded28e194"}
Nov 02 13:30:03.385087 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.385018    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls" event={"ID":"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921","Type":"ContainerStarted","Data":"f0382797fb81b7d654bccc109f826459e4c2d2e54843f0533f1979a9463219b7"}
Nov 02 13:30:03.506471 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.506405    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7" podStartSLOduration=1.506386258 podStartE2EDuration="1.506386258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:03.503450261 +0000 UTC m=+5632.067158672" watchObservedRunningTime="2026-11-02 13:30:03.506386258 +0000 UTC m=+5632.070094667"
Nov 02 13:30:03.596788 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.596753    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:03.626118 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.626014    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:03.630723 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:03.630516    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls" podStartSLOduration=1.630494774 podStartE2EDuration="1.630494774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:03.629258282 +0000 UTC m=+5632.192966717" watchObservedRunningTime="2026-11-02 13:30:03.630494774 +0000 UTC m=+5632.194203216"
Nov 02 13:30:04.412677 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:04.412626    4464 scope.go:122] "RemoveContainer" containerID="3c1f721f8ef108284e8f9678daa70bb939ad9f3be9f7b8fe77052cb260b45f95"
Nov 02 13:30:04.413254 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:30:04.412929    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893770-kc2k2_odh-ai-gateway-infra(6bba49fc-8f4f-4f61-9835-c4f51d9f5d09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2" podUID="6bba49fc-8f4f-4f61-9835-c4f51d9f5d09"
Nov 02 13:30:04.581803 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:04.581770    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-zfkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body=
Nov 02 13:30:04.582135 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:04.582093    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls" podUID="85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused"
Nov 02 13:30:04.627626 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:04.627592    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bp4h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Nov 02 13:30:04.627783 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:04.627645    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7" podUID="02c134c2-e2f8-4a16-a557-65786f86217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Nov 02 13:30:05.579999 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:05.576351    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-zfkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body=
Nov 02 13:30:05.579999 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:05.576413    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls" podUID="85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused"
Nov 02 13:30:05.626777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:05.626743    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bp4h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Nov 02 13:30:05.626946 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:05.626791    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7" podUID="02c134c2-e2f8-4a16-a557-65786f86217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Nov 02 13:30:06.580000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:06.575341    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-zfkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body=
Nov 02 13:30:06.580000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:06.575403    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls" podUID="85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused"
Nov 02 13:30:06.626059 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:06.626021    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bp4h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Nov 02 13:30:06.626239 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:06.626082    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7" podUID="02c134c2-e2f8-4a16-a557-65786f86217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Nov 02 13:30:07.576390 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:07.576352    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-zfkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body=
Nov 02 13:30:07.584101 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:07.584065    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls" podUID="85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused"
Nov 02 13:30:07.626620 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:07.626585    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bp4h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Nov 02 13:30:07.626766 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:07.626640    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7" podUID="02c134c2-e2f8-4a16-a557-65786f86217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Nov 02 13:30:08.576409 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:08.576369    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-zfkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body=
Nov 02 13:30:08.576698 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:08.576673    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls" podUID="85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused"
Nov 02 13:30:08.626378 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:08.626351    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bp4h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Nov 02 13:30:08.626831 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:08.626795    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7" podUID="02c134c2-e2f8-4a16-a557-65786f86217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Nov 02 13:30:09.575275 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:09.575235    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-zfkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body=
Nov 02 13:30:09.575499 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:09.575293    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls" podUID="85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused"
Nov 02 13:30:09.626668 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:09.626631    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bp4h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Nov 02 13:30:09.627172 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:09.626690    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7" podUID="02c134c2-e2f8-4a16-a557-65786f86217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Nov 02 13:30:10.574786 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:10.574744    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-zfkls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused" start-of-body=
Nov 02 13:30:10.574971 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:10.574802    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls" podUID="85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": dial tcp 10.133.1.48:15021: connect: connection refused"
Nov 02 13:30:10.627010 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:10.626968    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bp4h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Nov 02 13:30:10.627396 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:10.627030    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7" podUID="02c134c2-e2f8-4a16-a557-65786f86217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Nov 02 13:30:12.319968 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:12.319937    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:12.321023 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:12.320998    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:12.321431 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:12.321416    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:30:12.386426 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:12.386398    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:12.386836 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:12.386814    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:12.387571 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:12.387552    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:30:14.575047 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:14.575019    4464 scope.go:122] "RemoveContainer" containerID="3c1f721f8ef108284e8f9678daa70bb939ad9f3be9f7b8fe77052cb260b45f95"
Nov 02 13:30:14.949106 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:14.949081    4464 scope.go:122] "RemoveContainer" containerID="0675c9ecac3ab0b85683c8b017f09d598fd74b586756f49dd2a3dcf29ef73271"
Nov 02 13:30:14.956971 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:14.956944    4464 scope.go:122] "RemoveContainer" containerID="ab1ead3c69d9567749be2b388d4819f237a61f1e1e069cd247f411c7cd1e4634"
Nov 02 13:30:15.473928 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:15.473896    4464 generic.go:356] "Generic (PLEG): container finished" podID="6bba49fc-8f4f-4f61-9835-c4f51d9f5d09" containerID="85a0d4a4f7949717ee934f7336abf6b994bc1f1fd5806688460897cea942deec" exitCode=60
Nov 02 13:30:15.474108 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:15.473970    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2" event={"ID":"6bba49fc-8f4f-4f61-9835-c4f51d9f5d09","Type":"ContainerDied","Data":"85a0d4a4f7949717ee934f7336abf6b994bc1f1fd5806688460897cea942deec"}
Nov 02 13:30:15.474108 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:15.474046    4464 scope.go:122] "RemoveContainer" containerID="3c1f721f8ef108284e8f9678daa70bb939ad9f3be9f7b8fe77052cb260b45f95"
Nov 02 13:30:15.474329 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:15.474305    4464 scope.go:122] "RemoveContainer" containerID="85a0d4a4f7949717ee934f7336abf6b994bc1f1fd5806688460897cea942deec"
Nov 02 13:30:15.474548 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:30:15.474530    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893770-kc2k2_odh-ai-gateway-infra(6bba49fc-8f4f-4f61-9835-c4f51d9f5d09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2" podUID="6bba49fc-8f4f-4f61-9835-c4f51d9f5d09"
Nov 02 13:30:16.603331 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:16.603298    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2"]
Nov 02 13:30:16.738166 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:16.738143    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2"
Nov 02 13:30:16.828922 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:16.828891    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bba49fc-8f4f-4f61-9835-c4f51d9f5d09-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bba49fc-8f4f-4f61-9835-c4f51d9f5d09-maas-api-service-ca\") pod \"6bba49fc-8f4f-4f61-9835-c4f51d9f5d09\" (UID: \"6bba49fc-8f4f-4f61-9835-c4f51d9f5d09\") "
Nov 02 13:30:16.829109 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:16.828938    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bba49fc-8f4f-4f61-9835-c4f51d9f5d09-kube-api-access-dwsk7\" (UniqueName: \"kubernetes.io/projected/6bba49fc-8f4f-4f61-9835-c4f51d9f5d09-kube-api-access-dwsk7\") pod \"6bba49fc-8f4f-4f61-9835-c4f51d9f5d09\" (UID: \"6bba49fc-8f4f-4f61-9835-c4f51d9f5d09\") "
Nov 02 13:30:16.829268 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:16.829246    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bba49fc-8f4f-4f61-9835-c4f51d9f5d09-maas-api-service-ca" pod "6bba49fc-8f4f-4f61-9835-c4f51d9f5d09" (UID: "6bba49fc-8f4f-4f61-9835-c4f51d9f5d09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:16.831014 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:16.830988    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bba49fc-8f4f-4f61-9835-c4f51d9f5d09-kube-api-access-dwsk7" pod "6bba49fc-8f4f-4f61-9835-c4f51d9f5d09" (UID: "6bba49fc-8f4f-4f61-9835-c4f51d9f5d09"). InnerVolumeSpecName "kube-api-access-dwsk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:16.930105 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:16.930024    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bba49fc-8f4f-4f61-9835-c4f51d9f5d09-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:30:16.930105 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:16.930052    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwsk7\" (UniqueName: \"kubernetes.io/projected/6bba49fc-8f4f-4f61-9835-c4f51d9f5d09-kube-api-access-dwsk7\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:30:17.487122 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:17.487085    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2" event={"ID":"6bba49fc-8f4f-4f61-9835-c4f51d9f5d09","Type":"ContainerDied","Data":"33eefb95e63ab5517b69f8a73cb3b0a732ebe59d4445f36e8f5e8768991bd90f"}
Nov 02 13:30:17.487290 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:17.487128    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2"
Nov 02 13:30:17.487290 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:17.487156    4464 scope.go:122] "RemoveContainer" containerID="85a0d4a4f7949717ee934f7336abf6b994bc1f1fd5806688460897cea942deec"
Nov 02 13:30:17.653209 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:17.653180    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2"]
Nov 02 13:30:17.703800 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:17.703767    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893770-kc2k2"]
Nov 02 13:30:19.580383 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:30:19.580354    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bba49fc-8f4f-4f61-9835-c4f51d9f5d09" path="/var/lib/kubelet/pods/6bba49fc-8f4f-4f61-9835-c4f51d9f5d09/volumes"
Nov 02 13:30:28.291732 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:30:28.291693    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:30:58.291924 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:30:58.291883    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:31:08.244009 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.243966    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"]
Nov 02 13:31:08.244475 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.244435    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6bba49fc-8f4f-4f61-9835-c4f51d9f5d09" containerName="cleanup"
Nov 02 13:31:08.244475 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.244446    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bba49fc-8f4f-4f61-9835-c4f51d9f5d09" containerName="cleanup"
Nov 02 13:31:08.244475 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.244461    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6bba49fc-8f4f-4f61-9835-c4f51d9f5d09" containerName="cleanup"
Nov 02 13:31:08.244475 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.244466    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bba49fc-8f4f-4f61-9835-c4f51d9f5d09" containerName="cleanup"
Nov 02 13:31:08.244475 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.244475    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6bba49fc-8f4f-4f61-9835-c4f51d9f5d09" containerName="cleanup"
Nov 02 13:31:08.244652 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.244480    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bba49fc-8f4f-4f61-9835-c4f51d9f5d09" containerName="cleanup"
Nov 02 13:31:08.244652 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.244540    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bba49fc-8f4f-4f61-9835-c4f51d9f5d09" containerName="cleanup"
Nov 02 13:31:08.244652 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.244550    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bba49fc-8f4f-4f61-9835-c4f51d9f5d09" containerName="cleanup"
Nov 02 13:31:08.247476 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.247461    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"
Nov 02 13:31:08.353900 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.353862    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:31:08.353900 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.353862    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:31:08.353900 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.353890    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:31:08.354179 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.353907    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:31:08.356116 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.356097    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"]
Nov 02 13:31:08.408618 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.408588    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h64kq\" (UniqueName: \"kubernetes.io/projected/a21c63fd-6e08-467d-8460-5da00b64519a-kube-api-access-h64kq\") pod \"maas-api-revoke-sub-e2547622-xdhrs\" (UID: \"a21c63fd-6e08-467d-8460-5da00b64519a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"
Nov 02 13:31:08.408762 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.408697    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21c63fd-6e08-467d-8460-5da00b64519a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-xdhrs\" (UID: \"a21c63fd-6e08-467d-8460-5da00b64519a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"
Nov 02 13:31:08.408762 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.408730    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21c63fd-6e08-467d-8460-5da00b64519a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-xdhrs\" (UID: \"a21c63fd-6e08-467d-8460-5da00b64519a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"
Nov 02 13:31:08.509762 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.509693    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h64kq\" (UniqueName: \"kubernetes.io/projected/a21c63fd-6e08-467d-8460-5da00b64519a-kube-api-access-h64kq\") pod \"maas-api-revoke-sub-e2547622-xdhrs\" (UID: \"a21c63fd-6e08-467d-8460-5da00b64519a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"
Nov 02 13:31:08.509762 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.509755    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21c63fd-6e08-467d-8460-5da00b64519a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-xdhrs\" (UID: \"a21c63fd-6e08-467d-8460-5da00b64519a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"
Nov 02 13:31:08.509932 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.509788    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21c63fd-6e08-467d-8460-5da00b64519a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-xdhrs\" (UID: \"a21c63fd-6e08-467d-8460-5da00b64519a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"
Nov 02 13:31:08.510542 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.510514    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21c63fd-6e08-467d-8460-5da00b64519a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-xdhrs\" (UID: \"a21c63fd-6e08-467d-8460-5da00b64519a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"
Nov 02 13:31:08.512159 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.512141    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21c63fd-6e08-467d-8460-5da00b64519a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-xdhrs\" (UID: \"a21c63fd-6e08-467d-8460-5da00b64519a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"
Nov 02 13:31:08.575091 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.575063    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h64kq\" (UniqueName: \"kubernetes.io/projected/a21c63fd-6e08-467d-8460-5da00b64519a-kube-api-access-h64kq\") pod \"maas-api-revoke-sub-e2547622-xdhrs\" (UID: \"a21c63fd-6e08-467d-8460-5da00b64519a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"
Nov 02 13:31:08.860623 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:08.860539    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"
Nov 02 13:31:09.233277 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:09.233251    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"]
Nov 02 13:31:09.234179 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:31:09.234150    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21c63fd_6e08_467d_8460_5da00b64519a.slice/crio-73923f1f443cec25ca1b816d9708bca0e6b055d6cc9980b0c75c796c9e6ab894 WatchSource:0}: Error finding container 73923f1f443cec25ca1b816d9708bca0e6b055d6cc9980b0c75c796c9e6ab894: Status 404 returned error can't find the container with id 73923f1f443cec25ca1b816d9708bca0e6b055d6cc9980b0c75c796c9e6ab894
Nov 02 13:31:09.728048 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:09.728007    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs" event={"ID":"a21c63fd-6e08-467d-8460-5da00b64519a","Type":"ContainerStarted","Data":"4f97e527cb0c0a20e8aaa4f80b197ddf1fc8b7f372b5614bf830f428dea38608"}
Nov 02 13:31:09.728489 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:09.728072    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs" event={"ID":"a21c63fd-6e08-467d-8460-5da00b64519a","Type":"ContainerStarted","Data":"73923f1f443cec25ca1b816d9708bca0e6b055d6cc9980b0c75c796c9e6ab894"}
Nov 02 13:31:09.847216 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:09.847129    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs" podStartSLOduration=1.8471171119999998 podStartE2EDuration="1.847117112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:31:09.845510697 +0000 UTC m=+5698.409219109" watchObservedRunningTime="2026-11-02 13:31:09.847117112 +0000 UTC m=+5698.410825581"
Nov 02 13:31:10.734106 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:10.734074    4464 generic.go:356] "Generic (PLEG): container finished" podID="a21c63fd-6e08-467d-8460-5da00b64519a" containerID="4f97e527cb0c0a20e8aaa4f80b197ddf1fc8b7f372b5614bf830f428dea38608" exitCode=0
Nov 02 13:31:10.734510 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:10.734155    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs" event={"ID":"a21c63fd-6e08-467d-8460-5da00b64519a","Type":"ContainerDied","Data":"4f97e527cb0c0a20e8aaa4f80b197ddf1fc8b7f372b5614bf830f428dea38608"}
Nov 02 13:31:11.872772 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:11.872749    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"
Nov 02 13:31:12.042588 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:12.042511    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a21c63fd-6e08-467d-8460-5da00b64519a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21c63fd-6e08-467d-8460-5da00b64519a-maas-api-xks-ca\") pod \"a21c63fd-6e08-467d-8460-5da00b64519a\" (UID: \"a21c63fd-6e08-467d-8460-5da00b64519a\") "
Nov 02 13:31:12.042772 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:12.042640    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a21c63fd-6e08-467d-8460-5da00b64519a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21c63fd-6e08-467d-8460-5da00b64519a-maas-api-openshift-ca\") pod \"a21c63fd-6e08-467d-8460-5da00b64519a\" (UID: \"a21c63fd-6e08-467d-8460-5da00b64519a\") "
Nov 02 13:31:12.042772 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:12.042677    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a21c63fd-6e08-467d-8460-5da00b64519a-kube-api-access-h64kq\" (UniqueName: \"kubernetes.io/projected/a21c63fd-6e08-467d-8460-5da00b64519a-kube-api-access-h64kq\") pod \"a21c63fd-6e08-467d-8460-5da00b64519a\" (UID: \"a21c63fd-6e08-467d-8460-5da00b64519a\") "
Nov 02 13:31:12.043000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:12.042958    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a21c63fd-6e08-467d-8460-5da00b64519a-maas-api-openshift-ca" pod "a21c63fd-6e08-467d-8460-5da00b64519a" (UID: "a21c63fd-6e08-467d-8460-5da00b64519a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:31:12.044638 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:12.044616    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a21c63fd-6e08-467d-8460-5da00b64519a-maas-api-xks-ca" pod "a21c63fd-6e08-467d-8460-5da00b64519a" (UID: "a21c63fd-6e08-467d-8460-5da00b64519a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:31:12.044729 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:12.044690    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a21c63fd-6e08-467d-8460-5da00b64519a-kube-api-access-h64kq" pod "a21c63fd-6e08-467d-8460-5da00b64519a" (UID: "a21c63fd-6e08-467d-8460-5da00b64519a"). InnerVolumeSpecName "kube-api-access-h64kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:12.143969 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:12.143942    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a21c63fd-6e08-467d-8460-5da00b64519a-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:31:12.143969 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:12.143966    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h64kq\" (UniqueName: \"kubernetes.io/projected/a21c63fd-6e08-467d-8460-5da00b64519a-kube-api-access-h64kq\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:31:12.144126 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:12.143997    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a21c63fd-6e08-467d-8460-5da00b64519a-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:31:12.309481 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:12.309417    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:31:12.325987 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:12.325954    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:31:12.356171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:12.356151    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:31:12.374362 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:12.374340    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:31:12.751652 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:12.751622    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"
Nov 02 13:31:12.751852 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:12.751659    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs" event={"ID":"a21c63fd-6e08-467d-8460-5da00b64519a","Type":"ContainerDied","Data":"73923f1f443cec25ca1b816d9708bca0e6b055d6cc9980b0c75c796c9e6ab894"}
Nov 02 13:31:12.751852 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:12.751721    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73923f1f443cec25ca1b816d9708bca0e6b055d6cc9980b0c75c796c9e6ab894"
Nov 02 13:31:19.826781 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:19.826743    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"]
Nov 02 13:31:24.799274 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:24.799195    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:31:24.827815 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:24.827763    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2" podUID="17690f95-d8a7-4362-80ef-ddc3ae7ccca5" containerName="payload-processing" containerID="cri-o://5734a52213e68d506e5e1ec1e2820d6af03c903a5578d17d07d98aa04ab5ec9e" gracePeriod=25
Nov 02 13:31:25.078483 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.078451    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:31:25.165167 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.165136    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-kube-api-access-hs7w9\" (UniqueName: \"kubernetes.io/projected/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-kube-api-access-hs7w9\") pod \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\" (UID: \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\") "
Nov 02 13:31:25.165338 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.165238    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-routing-overlay-volume\") pod \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\" (UID: \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\") "
Nov 02 13:31:25.165338 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.165318    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-provider-credentials\") pod \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\" (UID: \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\") "
Nov 02 13:31:25.165437 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.165350    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-plugins-config-volume\") pod \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\" (UID: \"17690f95-d8a7-4362-80ef-ddc3ae7ccca5\") "
Nov 02 13:31:25.165730 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.165687    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-routing-overlay-volume" pod "17690f95-d8a7-4362-80ef-ddc3ae7ccca5" (UID: "17690f95-d8a7-4362-80ef-ddc3ae7ccca5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:31:25.165858 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.165741    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-plugins-config-volume" pod "17690f95-d8a7-4362-80ef-ddc3ae7ccca5" (UID: "17690f95-d8a7-4362-80ef-ddc3ae7ccca5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:31:25.167305 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.167283    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-kube-api-access-hs7w9" pod "17690f95-d8a7-4362-80ef-ddc3ae7ccca5" (UID: "17690f95-d8a7-4362-80ef-ddc3ae7ccca5"). InnerVolumeSpecName "kube-api-access-hs7w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:25.167411 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.167311    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-provider-credentials" pod "17690f95-d8a7-4362-80ef-ddc3ae7ccca5" (UID: "17690f95-d8a7-4362-80ef-ddc3ae7ccca5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:25.266433 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.266402    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:31:25.266433 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.266431    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs7w9\" (UniqueName: \"kubernetes.io/projected/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-kube-api-access-hs7w9\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:31:25.266433 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.266442    4464 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-routing-overlay-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:31:25.266687 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.266452    4464 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/17690f95-d8a7-4362-80ef-ddc3ae7ccca5-provider-credentials\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:31:25.826300 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.826256    4464 generic.go:356] "Generic (PLEG): container finished" podID="17690f95-d8a7-4362-80ef-ddc3ae7ccca5" containerID="5734a52213e68d506e5e1ec1e2820d6af03c903a5578d17d07d98aa04ab5ec9e" exitCode=0
Nov 02 13:31:25.826942 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.826299    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2" event={"ID":"17690f95-d8a7-4362-80ef-ddc3ae7ccca5","Type":"ContainerDied","Data":"5734a52213e68d506e5e1ec1e2820d6af03c903a5578d17d07d98aa04ab5ec9e"}
Nov 02 13:31:25.826942 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.826334    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"
Nov 02 13:31:25.826942 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.826369    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2" event={"ID":"17690f95-d8a7-4362-80ef-ddc3ae7ccca5","Type":"ContainerDied","Data":"9187844d227a5e39f63136aa8b8b74e6e63672aef4a47c42ab7b26b73c309c19"}
Nov 02 13:31:25.826942 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.826401    4464 scope.go:122] "RemoveContainer" containerID="5734a52213e68d506e5e1ec1e2820d6af03c903a5578d17d07d98aa04ab5ec9e"
Nov 02 13:31:25.835839 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.835814    4464 scope.go:122] "RemoveContainer" containerID="5734a52213e68d506e5e1ec1e2820d6af03c903a5578d17d07d98aa04ab5ec9e"
Nov 02 13:31:25.836165 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:31:25.836143    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5734a52213e68d506e5e1ec1e2820d6af03c903a5578d17d07d98aa04ab5ec9e\": container with ID starting with 5734a52213e68d506e5e1ec1e2820d6af03c903a5578d17d07d98aa04ab5ec9e not found: ID does not exist" containerID="5734a52213e68d506e5e1ec1e2820d6af03c903a5578d17d07d98aa04ab5ec9e"
Nov 02 13:31:25.836278 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.836172    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5734a52213e68d506e5e1ec1e2820d6af03c903a5578d17d07d98aa04ab5ec9e"} err="failed to get container status \"5734a52213e68d506e5e1ec1e2820d6af03c903a5578d17d07d98aa04ab5ec9e\": rpc error: code = NotFound desc = could not find container \"5734a52213e68d506e5e1ec1e2820d6af03c903a5578d17d07d98aa04ab5ec9e\": container with ID starting with 5734a52213e68d506e5e1ec1e2820d6af03c903a5578d17d07d98aa04ab5ec9e not found: ID does not exist"
Nov 02 13:31:25.996403 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:25.996366    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"]
Nov 02 13:31:26.045089 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:26.045053    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-x2xm2"]
Nov 02 13:31:27.593904 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:27.593860    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17690f95-d8a7-4362-80ef-ddc3ae7ccca5" path="/var/lib/kubelet/pods/17690f95-d8a7-4362-80ef-ddc3ae7ccca5/volumes"
Nov 02 13:31:28.319909 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:31:28.319832    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:31:28.362458 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:28.362123    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-zfkls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:31:28.362458 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:28.362191    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls" podUID="85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:31:28.429143 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:28.429099    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bp4h7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:31:28.429327 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:28.429151    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7" podUID="02c134c2-e2f8-4a16-a557-65786f86217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:31:33.984159 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:33.984126    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:31:40.966617 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:40.966576    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"]
Nov 02 13:31:40.967292 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:40.967011    4464 cpu_manager.go:414] "containerMap: removing container" podUID="a21c63fd-6e08-467d-8460-5da00b64519a" containerName="revoke-keys"
Nov 02 13:31:40.967292 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:40.967023    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a21c63fd-6e08-467d-8460-5da00b64519a" containerName="revoke-keys"
Nov 02 13:31:40.967292 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:40.967042    4464 cpu_manager.go:414] "containerMap: removing container" podUID="17690f95-d8a7-4362-80ef-ddc3ae7ccca5" containerName="payload-processing"
Nov 02 13:31:40.967292 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:40.967048    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17690f95-d8a7-4362-80ef-ddc3ae7ccca5" containerName="payload-processing"
Nov 02 13:31:40.967292 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:40.967119    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="a21c63fd-6e08-467d-8460-5da00b64519a" containerName="revoke-keys"
Nov 02 13:31:40.967292 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:40.967131    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="17690f95-d8a7-4362-80ef-ddc3ae7ccca5" containerName="payload-processing"
Nov 02 13:31:40.967292 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:40.967141    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bba49fc-8f4f-4f61-9835-c4f51d9f5d09" containerName="cleanup"
Nov 02 13:31:40.970356 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:40.970338    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"
Nov 02 13:31:41.071039 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:41.071005    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx5p4\" (UniqueName: \"kubernetes.io/projected/034e47cc-bfb4-4d60-a810-0f720700a2a2-kube-api-access-bx5p4\") pod \"maas-api-revoke-sub-5eeb23fc-lkgv4\" (UID: \"034e47cc-bfb4-4d60-a810-0f720700a2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"
Nov 02 13:31:41.071217 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:41.071051    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/034e47cc-bfb4-4d60-a810-0f720700a2a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eeb23fc-lkgv4\" (UID: \"034e47cc-bfb4-4d60-a810-0f720700a2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"
Nov 02 13:31:41.071217 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:41.071182    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/034e47cc-bfb4-4d60-a810-0f720700a2a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eeb23fc-lkgv4\" (UID: \"034e47cc-bfb4-4d60-a810-0f720700a2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"
Nov 02 13:31:41.075925 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:41.075896    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:31:41.076060 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:41.075952    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:31:41.076060 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:41.076054    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:31:41.076175 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:41.076125    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:31:41.078087 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:41.078059    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"]
Nov 02 13:31:41.172510 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:41.172478    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5p4\" (UniqueName: \"kubernetes.io/projected/034e47cc-bfb4-4d60-a810-0f720700a2a2-kube-api-access-bx5p4\") pod \"maas-api-revoke-sub-5eeb23fc-lkgv4\" (UID: \"034e47cc-bfb4-4d60-a810-0f720700a2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"
Nov 02 13:31:41.172685 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:41.172517    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/034e47cc-bfb4-4d60-a810-0f720700a2a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eeb23fc-lkgv4\" (UID: \"034e47cc-bfb4-4d60-a810-0f720700a2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"
Nov 02 13:31:41.172685 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:41.172603    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/034e47cc-bfb4-4d60-a810-0f720700a2a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eeb23fc-lkgv4\" (UID: \"034e47cc-bfb4-4d60-a810-0f720700a2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"
Nov 02 13:31:41.173248 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:41.173223    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/034e47cc-bfb4-4d60-a810-0f720700a2a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eeb23fc-lkgv4\" (UID: \"034e47cc-bfb4-4d60-a810-0f720700a2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"
Nov 02 13:31:41.175047 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:41.175029    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/034e47cc-bfb4-4d60-a810-0f720700a2a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eeb23fc-lkgv4\" (UID: \"034e47cc-bfb4-4d60-a810-0f720700a2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"
Nov 02 13:31:41.234778 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:41.234750    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5p4\" (UniqueName: \"kubernetes.io/projected/034e47cc-bfb4-4d60-a810-0f720700a2a2-kube-api-access-bx5p4\") pod \"maas-api-revoke-sub-5eeb23fc-lkgv4\" (UID: \"034e47cc-bfb4-4d60-a810-0f720700a2a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"
Nov 02 13:31:41.283773 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:41.283746    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"
Nov 02 13:31:41.656799 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:41.656773    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"]
Nov 02 13:31:41.658298 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:31:41.658269    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod034e47cc_bfb4_4d60_a810_0f720700a2a2.slice/crio-aa9c0d728582a805babce5bbe0f798c08b9a3e5f7fcbc79cc029e99f3fab84c5 WatchSource:0}: Error finding container aa9c0d728582a805babce5bbe0f798c08b9a3e5f7fcbc79cc029e99f3fab84c5: Status 404 returned error can't find the container with id aa9c0d728582a805babce5bbe0f798c08b9a3e5f7fcbc79cc029e99f3fab84c5
Nov 02 13:31:41.944660 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:41.944581    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4" event={"ID":"034e47cc-bfb4-4d60-a810-0f720700a2a2","Type":"ContainerStarted","Data":"aa9c0d728582a805babce5bbe0f798c08b9a3e5f7fcbc79cc029e99f3fab84c5"}
Nov 02 13:31:42.950689 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:42.950600    4464 generic.go:356] "Generic (PLEG): container finished" podID="034e47cc-bfb4-4d60-a810-0f720700a2a2" containerID="d469ca7c6fb7ce61a043650d9ba0b4692af7efa64bdbc3662d2d3f08463fec5d" exitCode=0
Nov 02 13:31:42.950689 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:42.950677    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4" event={"ID":"034e47cc-bfb4-4d60-a810-0f720700a2a2","Type":"ContainerDied","Data":"d469ca7c6fb7ce61a043650d9ba0b4692af7efa64bdbc3662d2d3f08463fec5d"}
Nov 02 13:31:44.096211 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:44.096173    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"
Nov 02 13:31:44.202093 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:44.202063    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/034e47cc-bfb4-4d60-a810-0f720700a2a2-kube-api-access-bx5p4\" (UniqueName: \"kubernetes.io/projected/034e47cc-bfb4-4d60-a810-0f720700a2a2-kube-api-access-bx5p4\") pod \"034e47cc-bfb4-4d60-a810-0f720700a2a2\" (UID: \"034e47cc-bfb4-4d60-a810-0f720700a2a2\") "
Nov 02 13:31:44.202264 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:44.202132    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/034e47cc-bfb4-4d60-a810-0f720700a2a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/034e47cc-bfb4-4d60-a810-0f720700a2a2-maas-api-openshift-ca\") pod \"034e47cc-bfb4-4d60-a810-0f720700a2a2\" (UID: \"034e47cc-bfb4-4d60-a810-0f720700a2a2\") "
Nov 02 13:31:44.202264 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:44.202193    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/034e47cc-bfb4-4d60-a810-0f720700a2a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/034e47cc-bfb4-4d60-a810-0f720700a2a2-maas-api-xks-ca\") pod \"034e47cc-bfb4-4d60-a810-0f720700a2a2\" (UID: \"034e47cc-bfb4-4d60-a810-0f720700a2a2\") "
Nov 02 13:31:44.202540 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:44.202516    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/034e47cc-bfb4-4d60-a810-0f720700a2a2-maas-api-openshift-ca" pod "034e47cc-bfb4-4d60-a810-0f720700a2a2" (UID: "034e47cc-bfb4-4d60-a810-0f720700a2a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:31:44.204131 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:44.204107    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/034e47cc-bfb4-4d60-a810-0f720700a2a2-maas-api-xks-ca" pod "034e47cc-bfb4-4d60-a810-0f720700a2a2" (UID: "034e47cc-bfb4-4d60-a810-0f720700a2a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:31:44.204260 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:44.204125    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/034e47cc-bfb4-4d60-a810-0f720700a2a2-kube-api-access-bx5p4" pod "034e47cc-bfb4-4d60-a810-0f720700a2a2" (UID: "034e47cc-bfb4-4d60-a810-0f720700a2a2"). InnerVolumeSpecName "kube-api-access-bx5p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:44.303243 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:44.303215    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/034e47cc-bfb4-4d60-a810-0f720700a2a2-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:31:44.303243 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:44.303241    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx5p4\" (UniqueName: \"kubernetes.io/projected/034e47cc-bfb4-4d60-a810-0f720700a2a2-kube-api-access-bx5p4\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:31:44.303429 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:44.303251    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/034e47cc-bfb4-4d60-a810-0f720700a2a2-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:31:44.960908 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:44.960874    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4" event={"ID":"034e47cc-bfb4-4d60-a810-0f720700a2a2","Type":"ContainerDied","Data":"aa9c0d728582a805babce5bbe0f798c08b9a3e5f7fcbc79cc029e99f3fab84c5"}
Nov 02 13:31:44.961197 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:44.960931    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa9c0d728582a805babce5bbe0f798c08b9a3e5f7fcbc79cc029e99f3fab84c5"
Nov 02 13:31:44.961197 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:44.960889    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"
Nov 02 13:31:54.114508 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:31:54.114473    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:31:58.290957 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:31:58.290916    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:32:03.592588 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:03.592556    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:32:05.298557 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.298520    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"]
Nov 02 13:32:05.299290 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.299253    4464 cpu_manager.go:414] "containerMap: removing container" podUID="034e47cc-bfb4-4d60-a810-0f720700a2a2" containerName="revoke-keys"
Nov 02 13:32:05.299290 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.299276    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="034e47cc-bfb4-4d60-a810-0f720700a2a2" containerName="revoke-keys"
Nov 02 13:32:05.299478 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.299367    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="034e47cc-bfb4-4d60-a810-0f720700a2a2" containerName="revoke-keys"
Nov 02 13:32:05.302499 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.302482    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"
Nov 02 13:32:05.357458 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.357332    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:32:05.357602 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.357329    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:32:05.357602 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.357399    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:32:05.408114 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.408086    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:32:05.410033 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.410014    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"]
Nov 02 13:32:05.416510 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.416489    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db2ebe5-57af-4f29-9cee-8c75db319b35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9mjc9\" (UID: \"6db2ebe5-57af-4f29-9cee-8c75db319b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"
Nov 02 13:32:05.416638 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.416621    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vfr9\" (UniqueName: \"kubernetes.io/projected/6db2ebe5-57af-4f29-9cee-8c75db319b35-kube-api-access-9vfr9\") pod \"maas-api-revoke-sub-2e1f5b5f-9mjc9\" (UID: \"6db2ebe5-57af-4f29-9cee-8c75db319b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"
Nov 02 13:32:05.416682 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.416655    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db2ebe5-57af-4f29-9cee-8c75db319b35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9mjc9\" (UID: \"6db2ebe5-57af-4f29-9cee-8c75db319b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"
Nov 02 13:32:05.517438 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.517410    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vfr9\" (UniqueName: \"kubernetes.io/projected/6db2ebe5-57af-4f29-9cee-8c75db319b35-kube-api-access-9vfr9\") pod \"maas-api-revoke-sub-2e1f5b5f-9mjc9\" (UID: \"6db2ebe5-57af-4f29-9cee-8c75db319b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"
Nov 02 13:32:05.517438 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.517441    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db2ebe5-57af-4f29-9cee-8c75db319b35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9mjc9\" (UID: \"6db2ebe5-57af-4f29-9cee-8c75db319b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"
Nov 02 13:32:05.517631 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.517487    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db2ebe5-57af-4f29-9cee-8c75db319b35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9mjc9\" (UID: \"6db2ebe5-57af-4f29-9cee-8c75db319b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"
Nov 02 13:32:05.518179 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.518154    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db2ebe5-57af-4f29-9cee-8c75db319b35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9mjc9\" (UID: \"6db2ebe5-57af-4f29-9cee-8c75db319b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"
Nov 02 13:32:05.519697 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.519679    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db2ebe5-57af-4f29-9cee-8c75db319b35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9mjc9\" (UID: \"6db2ebe5-57af-4f29-9cee-8c75db319b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"
Nov 02 13:32:05.580282 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.580222    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vfr9\" (UniqueName: \"kubernetes.io/projected/6db2ebe5-57af-4f29-9cee-8c75db319b35-kube-api-access-9vfr9\") pod \"maas-api-revoke-sub-2e1f5b5f-9mjc9\" (UID: \"6db2ebe5-57af-4f29-9cee-8c75db319b35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"
Nov 02 13:32:05.615559 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.615526    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"
Nov 02 13:32:05.984779 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:05.984753    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"]
Nov 02 13:32:05.986269 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:32:05.986248    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db2ebe5_57af_4f29_9cee_8c75db319b35.slice/crio-631e33eef5a71973c607c40c0bc881eba8cbb8f1faef94731d574563962b89dd WatchSource:0}: Error finding container 631e33eef5a71973c607c40c0bc881eba8cbb8f1faef94731d574563962b89dd: Status 404 returned error can't find the container with id 631e33eef5a71973c607c40c0bc881eba8cbb8f1faef94731d574563962b89dd
Nov 02 13:32:06.071275 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:06.071245    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9" event={"ID":"6db2ebe5-57af-4f29-9cee-8c75db319b35","Type":"ContainerStarted","Data":"631e33eef5a71973c607c40c0bc881eba8cbb8f1faef94731d574563962b89dd"}
Nov 02 13:32:07.077369 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:07.077292    4464 generic.go:356] "Generic (PLEG): container finished" podID="6db2ebe5-57af-4f29-9cee-8c75db319b35" containerID="9a11721b33a1498c233225766c9f9e0e18649f4da8fbc6a665cde19c14dc9b7a" exitCode=0
Nov 02 13:32:07.077724 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:07.077375    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9" event={"ID":"6db2ebe5-57af-4f29-9cee-8c75db319b35","Type":"ContainerDied","Data":"9a11721b33a1498c233225766c9f9e0e18649f4da8fbc6a665cde19c14dc9b7a"}
Nov 02 13:32:08.215858 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:08.215834    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"
Nov 02 13:32:08.345206 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:08.345124    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db2ebe5-57af-4f29-9cee-8c75db319b35-kube-api-access-9vfr9\" (UniqueName: \"kubernetes.io/projected/6db2ebe5-57af-4f29-9cee-8c75db319b35-kube-api-access-9vfr9\") pod \"6db2ebe5-57af-4f29-9cee-8c75db319b35\" (UID: \"6db2ebe5-57af-4f29-9cee-8c75db319b35\") "
Nov 02 13:32:08.345206 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:08.345193    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db2ebe5-57af-4f29-9cee-8c75db319b35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db2ebe5-57af-4f29-9cee-8c75db319b35-maas-api-openshift-ca\") pod \"6db2ebe5-57af-4f29-9cee-8c75db319b35\" (UID: \"6db2ebe5-57af-4f29-9cee-8c75db319b35\") "
Nov 02 13:32:08.345421 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:08.345253    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db2ebe5-57af-4f29-9cee-8c75db319b35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db2ebe5-57af-4f29-9cee-8c75db319b35-maas-api-xks-ca\") pod \"6db2ebe5-57af-4f29-9cee-8c75db319b35\" (UID: \"6db2ebe5-57af-4f29-9cee-8c75db319b35\") "
Nov 02 13:32:08.345558 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:08.345525    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db2ebe5-57af-4f29-9cee-8c75db319b35-maas-api-openshift-ca" pod "6db2ebe5-57af-4f29-9cee-8c75db319b35" (UID: "6db2ebe5-57af-4f29-9cee-8c75db319b35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:32:08.347267 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:08.347239    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db2ebe5-57af-4f29-9cee-8c75db319b35-maas-api-xks-ca" pod "6db2ebe5-57af-4f29-9cee-8c75db319b35" (UID: "6db2ebe5-57af-4f29-9cee-8c75db319b35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:32:08.347370 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:08.347337    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db2ebe5-57af-4f29-9cee-8c75db319b35-kube-api-access-9vfr9" pod "6db2ebe5-57af-4f29-9cee-8c75db319b35" (UID: "6db2ebe5-57af-4f29-9cee-8c75db319b35"). InnerVolumeSpecName "kube-api-access-9vfr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:08.446040 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:08.446010    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vfr9\" (UniqueName: \"kubernetes.io/projected/6db2ebe5-57af-4f29-9cee-8c75db319b35-kube-api-access-9vfr9\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:32:08.446040 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:08.446036    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db2ebe5-57af-4f29-9cee-8c75db319b35-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:32:08.446040 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:08.446046    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db2ebe5-57af-4f29-9cee-8c75db319b35-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:32:09.088343 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:09.088304    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"
Nov 02 13:32:09.088568 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:09.088357    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9" event={"ID":"6db2ebe5-57af-4f29-9cee-8c75db319b35","Type":"ContainerDied","Data":"631e33eef5a71973c607c40c0bc881eba8cbb8f1faef94731d574563962b89dd"}
Nov 02 13:32:09.088568 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:09.088402    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="631e33eef5a71973c607c40c0bc881eba8cbb8f1faef94731d574563962b89dd"
Nov 02 13:32:17.739644 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.739604    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"]
Nov 02 13:32:17.740375 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.740354    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6db2ebe5-57af-4f29-9cee-8c75db319b35" containerName="revoke-keys"
Nov 02 13:32:17.740424 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.740378    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db2ebe5-57af-4f29-9cee-8c75db319b35" containerName="revoke-keys"
Nov 02 13:32:17.740494 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.740482    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db2ebe5-57af-4f29-9cee-8c75db319b35" containerName="revoke-keys"
Nov 02 13:32:17.745331 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.745307    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"
Nov 02 13:32:17.800580 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.800545    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"]
Nov 02 13:32:17.836616 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.836587    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b587cc3f-12fe-4d07-a51c-a14f1b20b581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9wn6g\" (UID: \"b587cc3f-12fe-4d07-a51c-a14f1b20b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"
Nov 02 13:32:17.836765 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.836661    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b587cc3f-12fe-4d07-a51c-a14f1b20b581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9wn6g\" (UID: \"b587cc3f-12fe-4d07-a51c-a14f1b20b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"
Nov 02 13:32:17.836820 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.836771    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttg6m\" (UniqueName: \"kubernetes.io/projected/b587cc3f-12fe-4d07-a51c-a14f1b20b581-kube-api-access-ttg6m\") pod \"maas-api-revoke-sub-ec843018-9wn6g\" (UID: \"b587cc3f-12fe-4d07-a51c-a14f1b20b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"
Nov 02 13:32:17.847555 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.847537    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:32:17.847642 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.847562    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:32:17.847642 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.847606    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:32:17.849205 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.849191    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:32:17.937895 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.937868    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b587cc3f-12fe-4d07-a51c-a14f1b20b581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9wn6g\" (UID: \"b587cc3f-12fe-4d07-a51c-a14f1b20b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"
Nov 02 13:32:17.938092 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.937932    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttg6m\" (UniqueName: \"kubernetes.io/projected/b587cc3f-12fe-4d07-a51c-a14f1b20b581-kube-api-access-ttg6m\") pod \"maas-api-revoke-sub-ec843018-9wn6g\" (UID: \"b587cc3f-12fe-4d07-a51c-a14f1b20b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"
Nov 02 13:32:17.938092 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.937993    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b587cc3f-12fe-4d07-a51c-a14f1b20b581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9wn6g\" (UID: \"b587cc3f-12fe-4d07-a51c-a14f1b20b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"
Nov 02 13:32:17.938625 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.938599    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b587cc3f-12fe-4d07-a51c-a14f1b20b581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-9wn6g\" (UID: \"b587cc3f-12fe-4d07-a51c-a14f1b20b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"
Nov 02 13:32:17.940237 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:17.940216    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b587cc3f-12fe-4d07-a51c-a14f1b20b581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-9wn6g\" (UID: \"b587cc3f-12fe-4d07-a51c-a14f1b20b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"
Nov 02 13:32:18.000261 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:18.000205    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttg6m\" (UniqueName: \"kubernetes.io/projected/b587cc3f-12fe-4d07-a51c-a14f1b20b581-kube-api-access-ttg6m\") pod \"maas-api-revoke-sub-ec843018-9wn6g\" (UID: \"b587cc3f-12fe-4d07-a51c-a14f1b20b581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"
Nov 02 13:32:18.058710 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:18.058681    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"
Nov 02 13:32:18.189165 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:18.189135    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:32:18.430103 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:18.429998    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"]
Nov 02 13:32:18.432573 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:32:18.432545    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb587cc3f_12fe_4d07_a51c_a14f1b20b581.slice/crio-b9b59757e7bbe91bb7c35e528c7ea62f9adc7064af22c1e5e06dfd36213cce39 WatchSource:0}: Error finding container b9b59757e7bbe91bb7c35e528c7ea62f9adc7064af22c1e5e06dfd36213cce39: Status 404 returned error can't find the container with id b9b59757e7bbe91bb7c35e528c7ea62f9adc7064af22c1e5e06dfd36213cce39
Nov 02 13:32:19.147069 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:19.146952    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g" event={"ID":"b587cc3f-12fe-4d07-a51c-a14f1b20b581","Type":"ContainerStarted","Data":"f177c825675244cbe806b558fb63a652b53c09975b81db7cba5894f7864c4275"}
Nov 02 13:32:19.147430 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:19.147078    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g" event={"ID":"b587cc3f-12fe-4d07-a51c-a14f1b20b581","Type":"ContainerStarted","Data":"b9b59757e7bbe91bb7c35e528c7ea62f9adc7064af22c1e5e06dfd36213cce39"}
Nov 02 13:32:19.266554 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:19.266506    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g" podStartSLOduration=2.266493144 podStartE2EDuration="2.266493144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:32:19.264492505 +0000 UTC m=+5767.828200949" watchObservedRunningTime="2026-11-02 13:32:19.266493144 +0000 UTC m=+5767.830201547"
Nov 02 13:32:20.153565 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:20.153526    4464 generic.go:356] "Generic (PLEG): container finished" podID="b587cc3f-12fe-4d07-a51c-a14f1b20b581" containerID="f177c825675244cbe806b558fb63a652b53c09975b81db7cba5894f7864c4275" exitCode=0
Nov 02 13:32:20.154028 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:20.153612    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g" event={"ID":"b587cc3f-12fe-4d07-a51c-a14f1b20b581","Type":"ContainerDied","Data":"f177c825675244cbe806b558fb63a652b53c09975b81db7cba5894f7864c4275"}
Nov 02 13:32:21.289418 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:21.289388    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"
Nov 02 13:32:21.474058 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:21.474035    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b587cc3f-12fe-4d07-a51c-a14f1b20b581-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b587cc3f-12fe-4d07-a51c-a14f1b20b581-maas-api-xks-ca\") pod \"b587cc3f-12fe-4d07-a51c-a14f1b20b581\" (UID: \"b587cc3f-12fe-4d07-a51c-a14f1b20b581\") "
Nov 02 13:32:21.474244 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:21.474069    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b587cc3f-12fe-4d07-a51c-a14f1b20b581-kube-api-access-ttg6m\" (UniqueName: \"kubernetes.io/projected/b587cc3f-12fe-4d07-a51c-a14f1b20b581-kube-api-access-ttg6m\") pod \"b587cc3f-12fe-4d07-a51c-a14f1b20b581\" (UID: \"b587cc3f-12fe-4d07-a51c-a14f1b20b581\") "
Nov 02 13:32:21.474244 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:21.474106    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b587cc3f-12fe-4d07-a51c-a14f1b20b581-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b587cc3f-12fe-4d07-a51c-a14f1b20b581-maas-api-openshift-ca\") pod \"b587cc3f-12fe-4d07-a51c-a14f1b20b581\" (UID: \"b587cc3f-12fe-4d07-a51c-a14f1b20b581\") "
Nov 02 13:32:21.474553 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:21.474532    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b587cc3f-12fe-4d07-a51c-a14f1b20b581-maas-api-openshift-ca" pod "b587cc3f-12fe-4d07-a51c-a14f1b20b581" (UID: "b587cc3f-12fe-4d07-a51c-a14f1b20b581"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:32:21.476247 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:21.476209    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b587cc3f-12fe-4d07-a51c-a14f1b20b581-maas-api-xks-ca" pod "b587cc3f-12fe-4d07-a51c-a14f1b20b581" (UID: "b587cc3f-12fe-4d07-a51c-a14f1b20b581"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:32:21.476362 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:21.476251    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b587cc3f-12fe-4d07-a51c-a14f1b20b581-kube-api-access-ttg6m" pod "b587cc3f-12fe-4d07-a51c-a14f1b20b581" (UID: "b587cc3f-12fe-4d07-a51c-a14f1b20b581"). InnerVolumeSpecName "kube-api-access-ttg6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:21.575139 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:21.575114    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b587cc3f-12fe-4d07-a51c-a14f1b20b581-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:32:21.575249 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:21.575141    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttg6m\" (UniqueName: \"kubernetes.io/projected/b587cc3f-12fe-4d07-a51c-a14f1b20b581-kube-api-access-ttg6m\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:32:21.575249 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:21.575155    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b587cc3f-12fe-4d07-a51c-a14f1b20b581-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:32:22.165669 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:22.165637    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g" event={"ID":"b587cc3f-12fe-4d07-a51c-a14f1b20b581","Type":"ContainerDied","Data":"b9b59757e7bbe91bb7c35e528c7ea62f9adc7064af22c1e5e06dfd36213cce39"}
Nov 02 13:32:22.165669 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:22.165658    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"
Nov 02 13:32:22.165883 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:22.165689    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9b59757e7bbe91bb7c35e528c7ea62f9adc7064af22c1e5e06dfd36213cce39"
Nov 02 13:32:28.290582 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:32:28.290546    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:32:36.489819 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:36.489785    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:32:47.049178 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:47.049141    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"]
Nov 02 13:32:47.099649 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:47.099611    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af65f8a9-llxth"]
Nov 02 13:32:47.581027 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:47.580965    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="539ba697-a2eb-4f4d-b430-3347d0645c23" path="/var/lib/kubelet/pods/539ba697-a2eb-4f4d-b430-3347d0645c23/volumes"
Nov 02 13:32:55.555590 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.555510    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"]
Nov 02 13:32:55.555953 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.555944    4464 cpu_manager.go:414] "containerMap: removing container" podUID="b587cc3f-12fe-4d07-a51c-a14f1b20b581" containerName="revoke-keys"
Nov 02 13:32:55.556036 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.555954    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b587cc3f-12fe-4d07-a51c-a14f1b20b581" containerName="revoke-keys"
Nov 02 13:32:55.556079 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.556069    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="b587cc3f-12fe-4d07-a51c-a14f1b20b581" containerName="revoke-keys"
Nov 02 13:32:55.560295 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.560278    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"
Nov 02 13:32:55.665933 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.665902    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:32:55.666157 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.666028    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:32:55.666157 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.666028    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:32:55.666157 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.666083    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:32:55.668117 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.668096    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"]
Nov 02 13:32:55.711474 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.711448    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl7hv\" (UniqueName: \"kubernetes.io/projected/366dc96f-3d76-4610-93e9-2813242060ff-kube-api-access-bl7hv\") pod \"maas-api-revoke-sub-046e1343-lmmwr\" (UID: \"366dc96f-3d76-4610-93e9-2813242060ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"
Nov 02 13:32:55.711591 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.711504    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366dc96f-3d76-4610-93e9-2813242060ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-lmmwr\" (UID: \"366dc96f-3d76-4610-93e9-2813242060ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"
Nov 02 13:32:55.711767 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.711741    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366dc96f-3d76-4610-93e9-2813242060ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-lmmwr\" (UID: \"366dc96f-3d76-4610-93e9-2813242060ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"
Nov 02 13:32:55.812775 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.812706    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl7hv\" (UniqueName: \"kubernetes.io/projected/366dc96f-3d76-4610-93e9-2813242060ff-kube-api-access-bl7hv\") pod \"maas-api-revoke-sub-046e1343-lmmwr\" (UID: \"366dc96f-3d76-4610-93e9-2813242060ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"
Nov 02 13:32:55.812775 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.812754    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366dc96f-3d76-4610-93e9-2813242060ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-lmmwr\" (UID: \"366dc96f-3d76-4610-93e9-2813242060ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"
Nov 02 13:32:55.813039 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.812818    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366dc96f-3d76-4610-93e9-2813242060ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-lmmwr\" (UID: \"366dc96f-3d76-4610-93e9-2813242060ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"
Nov 02 13:32:55.813407 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.813385    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366dc96f-3d76-4610-93e9-2813242060ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-lmmwr\" (UID: \"366dc96f-3d76-4610-93e9-2813242060ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"
Nov 02 13:32:55.815113 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.815087    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366dc96f-3d76-4610-93e9-2813242060ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-lmmwr\" (UID: \"366dc96f-3d76-4610-93e9-2813242060ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"
Nov 02 13:32:55.881259 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:55.881232    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl7hv\" (UniqueName: \"kubernetes.io/projected/366dc96f-3d76-4610-93e9-2813242060ff-kube-api-access-bl7hv\") pod \"maas-api-revoke-sub-046e1343-lmmwr\" (UID: \"366dc96f-3d76-4610-93e9-2813242060ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"
Nov 02 13:32:56.172419 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:56.172342    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"
Nov 02 13:32:56.542619 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:56.542588    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"]
Nov 02 13:32:56.544527 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:32:56.544491    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366dc96f_3d76_4610_93e9_2813242060ff.slice/crio-57eb74844e3a0c5974669f725976c1d9e6217616a98cc2ba4a7cdfba0ab88490 WatchSource:0}: Error finding container 57eb74844e3a0c5974669f725976c1d9e6217616a98cc2ba4a7cdfba0ab88490: Status 404 returned error can't find the container with id 57eb74844e3a0c5974669f725976c1d9e6217616a98cc2ba4a7cdfba0ab88490
Nov 02 13:32:57.339040 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:57.339002    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr" event={"ID":"366dc96f-3d76-4610-93e9-2813242060ff","Type":"ContainerStarted","Data":"e69a39caa238347be560f57543947fa9ac79b53f7ae63e57a9d2a6e73a3c8f4e"}
Nov 02 13:32:57.339422 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:57.339067    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr" event={"ID":"366dc96f-3d76-4610-93e9-2813242060ff","Type":"ContainerStarted","Data":"57eb74844e3a0c5974669f725976c1d9e6217616a98cc2ba4a7cdfba0ab88490"}
Nov 02 13:32:57.457948 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:57.457896    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr" podStartSLOduration=2.457882183 podStartE2EDuration="2.457882183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:32:57.45675588 +0000 UTC m=+5806.020464292" watchObservedRunningTime="2026-11-02 13:32:57.457882183 +0000 UTC m=+5806.021590592"
Nov 02 13:32:58.291268 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:32:58.291233    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:32:58.345194 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:58.345167    4464 generic.go:356] "Generic (PLEG): container finished" podID="366dc96f-3d76-4610-93e9-2813242060ff" containerID="e69a39caa238347be560f57543947fa9ac79b53f7ae63e57a9d2a6e73a3c8f4e" exitCode=0
Nov 02 13:32:58.345553 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:58.345249    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr" event={"ID":"366dc96f-3d76-4610-93e9-2813242060ff","Type":"ContainerDied","Data":"e69a39caa238347be560f57543947fa9ac79b53f7ae63e57a9d2a6e73a3c8f4e"}
Nov 02 13:32:59.479879 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:59.479854    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"
Nov 02 13:32:59.646014 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:59.645940    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/366dc96f-3d76-4610-93e9-2813242060ff-kube-api-access-bl7hv\" (UniqueName: \"kubernetes.io/projected/366dc96f-3d76-4610-93e9-2813242060ff-kube-api-access-bl7hv\") pod \"366dc96f-3d76-4610-93e9-2813242060ff\" (UID: \"366dc96f-3d76-4610-93e9-2813242060ff\") "
Nov 02 13:32:59.646014 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:59.645995    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/366dc96f-3d76-4610-93e9-2813242060ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366dc96f-3d76-4610-93e9-2813242060ff-maas-api-openshift-ca\") pod \"366dc96f-3d76-4610-93e9-2813242060ff\" (UID: \"366dc96f-3d76-4610-93e9-2813242060ff\") "
Nov 02 13:32:59.646234 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:59.646130    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/366dc96f-3d76-4610-93e9-2813242060ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366dc96f-3d76-4610-93e9-2813242060ff-maas-api-xks-ca\") pod \"366dc96f-3d76-4610-93e9-2813242060ff\" (UID: \"366dc96f-3d76-4610-93e9-2813242060ff\") "
Nov 02 13:32:59.646353 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:59.646332    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/366dc96f-3d76-4610-93e9-2813242060ff-maas-api-openshift-ca" pod "366dc96f-3d76-4610-93e9-2813242060ff" (UID: "366dc96f-3d76-4610-93e9-2813242060ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:32:59.646521 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:59.646503    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366dc96f-3d76-4610-93e9-2813242060ff-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:32:59.648137 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:59.648118    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366dc96f-3d76-4610-93e9-2813242060ff-kube-api-access-bl7hv" pod "366dc96f-3d76-4610-93e9-2813242060ff" (UID: "366dc96f-3d76-4610-93e9-2813242060ff"). InnerVolumeSpecName "kube-api-access-bl7hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:59.648228 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:59.648187    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/366dc96f-3d76-4610-93e9-2813242060ff-maas-api-xks-ca" pod "366dc96f-3d76-4610-93e9-2813242060ff" (UID: "366dc96f-3d76-4610-93e9-2813242060ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:32:59.747271 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:59.747242    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl7hv\" (UniqueName: \"kubernetes.io/projected/366dc96f-3d76-4610-93e9-2813242060ff-kube-api-access-bl7hv\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:32:59.747271 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:32:59.747267    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366dc96f-3d76-4610-93e9-2813242060ff-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:33:00.356393 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:00.356363    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"
Nov 02 13:33:00.356589 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:00.356357    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr" event={"ID":"366dc96f-3d76-4610-93e9-2813242060ff","Type":"ContainerDied","Data":"57eb74844e3a0c5974669f725976c1d9e6217616a98cc2ba4a7cdfba0ab88490"}
Nov 02 13:33:00.356589 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:00.356493    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57eb74844e3a0c5974669f725976c1d9e6217616a98cc2ba4a7cdfba0ab88490"
Nov 02 13:33:06.324493 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.324460    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"]
Nov 02 13:33:06.324936 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.324920    4464 cpu_manager.go:414] "containerMap: removing container" podUID="366dc96f-3d76-4610-93e9-2813242060ff" containerName="revoke-keys"
Nov 02 13:33:06.325017 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.324937    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="366dc96f-3d76-4610-93e9-2813242060ff" containerName="revoke-keys"
Nov 02 13:33:06.325061 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.325029    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="366dc96f-3d76-4610-93e9-2813242060ff" containerName="revoke-keys"
Nov 02 13:33:06.328123 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.328107    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"
Nov 02 13:33:06.383000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.382955    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:33:06.383148 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.383044    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:33:06.408365 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.408336    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g92zz\" (UniqueName: \"kubernetes.io/projected/83b9779f-add3-47df-a72f-f2430fe99fd9-kube-api-access-g92zz\") pod \"maas-api-revoke-sub-d3fba056-zlh69\" (UID: \"83b9779f-add3-47df-a72f-f2430fe99fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"
Nov 02 13:33:06.408534 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.408372    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b9779f-add3-47df-a72f-f2430fe99fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-zlh69\" (UID: \"83b9779f-add3-47df-a72f-f2430fe99fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"
Nov 02 13:33:06.408534 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.408394    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b9779f-add3-47df-a72f-f2430fe99fd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-zlh69\" (UID: \"83b9779f-add3-47df-a72f-f2430fe99fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"
Nov 02 13:33:06.433786 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.433760    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:33:06.433786 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.433776    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:33:06.436052 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.436034    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"]
Nov 02 13:33:06.509721 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.509700    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g92zz\" (UniqueName: \"kubernetes.io/projected/83b9779f-add3-47df-a72f-f2430fe99fd9-kube-api-access-g92zz\") pod \"maas-api-revoke-sub-d3fba056-zlh69\" (UID: \"83b9779f-add3-47df-a72f-f2430fe99fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"
Nov 02 13:33:06.509866 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.509729    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b9779f-add3-47df-a72f-f2430fe99fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-zlh69\" (UID: \"83b9779f-add3-47df-a72f-f2430fe99fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"
Nov 02 13:33:06.509866 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.509752    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b9779f-add3-47df-a72f-f2430fe99fd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-zlh69\" (UID: \"83b9779f-add3-47df-a72f-f2430fe99fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"
Nov 02 13:33:06.510481 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.510461    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b9779f-add3-47df-a72f-f2430fe99fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-zlh69\" (UID: \"83b9779f-add3-47df-a72f-f2430fe99fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"
Nov 02 13:33:06.512038 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.512022    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b9779f-add3-47df-a72f-f2430fe99fd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-zlh69\" (UID: \"83b9779f-add3-47df-a72f-f2430fe99fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"
Nov 02 13:33:06.571301 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.571279    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g92zz\" (UniqueName: \"kubernetes.io/projected/83b9779f-add3-47df-a72f-f2430fe99fd9-kube-api-access-g92zz\") pod \"maas-api-revoke-sub-d3fba056-zlh69\" (UID: \"83b9779f-add3-47df-a72f-f2430fe99fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"
Nov 02 13:33:06.641074 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:06.641017    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"
Nov 02 13:33:07.013924 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:07.013892    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"]
Nov 02 13:33:07.014754 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:33:07.014728    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b9779f_add3_47df_a72f_f2430fe99fd9.slice/crio-2b0f9a40df0ad4e49e9d1b8ca5869212f28c8e6ec6a28593821d1bc5d50b2eae WatchSource:0}: Error finding container 2b0f9a40df0ad4e49e9d1b8ca5869212f28c8e6ec6a28593821d1bc5d50b2eae: Status 404 returned error can't find the container with id 2b0f9a40df0ad4e49e9d1b8ca5869212f28c8e6ec6a28593821d1bc5d50b2eae
Nov 02 13:33:07.394461 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:07.394395    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69" event={"ID":"83b9779f-add3-47df-a72f-f2430fe99fd9","Type":"ContainerStarted","Data":"80f0c8bf5ace0e9b3a5693eaf79d08636d8cdfcee3bbfa5870bd545601623457"}
Nov 02 13:33:07.394461 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:07.394447    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69" event={"ID":"83b9779f-add3-47df-a72f-f2430fe99fd9","Type":"ContainerStarted","Data":"2b0f9a40df0ad4e49e9d1b8ca5869212f28c8e6ec6a28593821d1bc5d50b2eae"}
Nov 02 13:33:07.512903 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:07.512862    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69" podStartSLOduration=1.512850333 podStartE2EDuration="1.512850333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:33:07.512164311 +0000 UTC m=+5816.075872722" watchObservedRunningTime="2026-11-02 13:33:07.512850333 +0000 UTC m=+5816.076558742"
Nov 02 13:33:08.400504 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:08.400469    4464 generic.go:356] "Generic (PLEG): container finished" podID="83b9779f-add3-47df-a72f-f2430fe99fd9" containerID="80f0c8bf5ace0e9b3a5693eaf79d08636d8cdfcee3bbfa5870bd545601623457" exitCode=0
Nov 02 13:33:08.400894 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:08.400552    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69" event={"ID":"83b9779f-add3-47df-a72f-f2430fe99fd9","Type":"ContainerDied","Data":"80f0c8bf5ace0e9b3a5693eaf79d08636d8cdfcee3bbfa5870bd545601623457"}
Nov 02 13:33:08.416116 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:08.416090    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:33:09.576316 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:09.576286    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"
Nov 02 13:33:09.743850 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:09.743800    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83b9779f-add3-47df-a72f-f2430fe99fd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b9779f-add3-47df-a72f-f2430fe99fd9-maas-api-openshift-ca\") pod \"83b9779f-add3-47df-a72f-f2430fe99fd9\" (UID: \"83b9779f-add3-47df-a72f-f2430fe99fd9\") "
Nov 02 13:33:09.744086 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:09.743881    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83b9779f-add3-47df-a72f-f2430fe99fd9-kube-api-access-g92zz\" (UniqueName: \"kubernetes.io/projected/83b9779f-add3-47df-a72f-f2430fe99fd9-kube-api-access-g92zz\") pod \"83b9779f-add3-47df-a72f-f2430fe99fd9\" (UID: \"83b9779f-add3-47df-a72f-f2430fe99fd9\") "
Nov 02 13:33:09.744086 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:09.743913    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83b9779f-add3-47df-a72f-f2430fe99fd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b9779f-add3-47df-a72f-f2430fe99fd9-maas-api-xks-ca\") pod \"83b9779f-add3-47df-a72f-f2430fe99fd9\" (UID: \"83b9779f-add3-47df-a72f-f2430fe99fd9\") "
Nov 02 13:33:09.744257 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:09.744225    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b9779f-add3-47df-a72f-f2430fe99fd9-maas-api-openshift-ca" pod "83b9779f-add3-47df-a72f-f2430fe99fd9" (UID: "83b9779f-add3-47df-a72f-f2430fe99fd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:33:09.744895 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:09.744862    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83b9779f-add3-47df-a72f-f2430fe99fd9-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:33:09.746587 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:09.746556    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83b9779f-add3-47df-a72f-f2430fe99fd9-maas-api-xks-ca" pod "83b9779f-add3-47df-a72f-f2430fe99fd9" (UID: "83b9779f-add3-47df-a72f-f2430fe99fd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:33:09.746713 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:09.746659    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b9779f-add3-47df-a72f-f2430fe99fd9-kube-api-access-g92zz" pod "83b9779f-add3-47df-a72f-f2430fe99fd9" (UID: "83b9779f-add3-47df-a72f-f2430fe99fd9"). InnerVolumeSpecName "kube-api-access-g92zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:09.846147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:09.846107    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g92zz\" (UniqueName: \"kubernetes.io/projected/83b9779f-add3-47df-a72f-f2430fe99fd9-kube-api-access-g92zz\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:33:09.846147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:09.846144    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83b9779f-add3-47df-a72f-f2430fe99fd9-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:33:10.440006 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:10.436795    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69" event={"ID":"83b9779f-add3-47df-a72f-f2430fe99fd9","Type":"ContainerDied","Data":"2b0f9a40df0ad4e49e9d1b8ca5869212f28c8e6ec6a28593821d1bc5d50b2eae"}
Nov 02 13:33:10.440006 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:10.436859    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b0f9a40df0ad4e49e9d1b8ca5869212f28c8e6ec6a28593821d1bc5d50b2eae"
Nov 02 13:33:10.440006 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:10.436931    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"
Nov 02 13:33:11.127742 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:11.127699    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gt29z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:33:11.128518 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:11.128476    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podUID="bf09d973-3e5b-4daa-aa7f-cf839b61d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:33:21.075217 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:21.075178    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:33:28.352694 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:33:28.352653    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:33:30.283724 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:30.283685    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:33:41.038109 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.038078    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"]
Nov 02 13:33:41.038543 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.038529    4464 cpu_manager.go:414] "containerMap: removing container" podUID="83b9779f-add3-47df-a72f-f2430fe99fd9" containerName="revoke-keys"
Nov 02 13:33:41.038605 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.038544    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83b9779f-add3-47df-a72f-f2430fe99fd9" containerName="revoke-keys"
Nov 02 13:33:41.038661 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.038613    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="83b9779f-add3-47df-a72f-f2430fe99fd9" containerName="revoke-keys"
Nov 02 13:33:41.041933 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.041915    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"
Nov 02 13:33:41.097274 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.097239    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:33:41.097470 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.097280    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:33:41.136268 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.136231    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83b973c-2a27-4338-b700-0a98903f17da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6lvj6\" (UID: \"b83b973c-2a27-4338-b700-0a98903f17da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"
Nov 02 13:33:41.136422 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.136380    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlgn5\" (UniqueName: \"kubernetes.io/projected/b83b973c-2a27-4338-b700-0a98903f17da-kube-api-access-rlgn5\") pod \"maas-api-revoke-sub-ae594979-6lvj6\" (UID: \"b83b973c-2a27-4338-b700-0a98903f17da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"
Nov 02 13:33:41.136471 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.136462    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83b973c-2a27-4338-b700-0a98903f17da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6lvj6\" (UID: \"b83b973c-2a27-4338-b700-0a98903f17da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"
Nov 02 13:33:41.147564 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.147548    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:33:41.147639 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.147576    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:33:41.149796 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.149781    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"]
Nov 02 13:33:41.237156 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.237130    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83b973c-2a27-4338-b700-0a98903f17da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6lvj6\" (UID: \"b83b973c-2a27-4338-b700-0a98903f17da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"
Nov 02 13:33:41.237332 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.237186    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlgn5\" (UniqueName: \"kubernetes.io/projected/b83b973c-2a27-4338-b700-0a98903f17da-kube-api-access-rlgn5\") pod \"maas-api-revoke-sub-ae594979-6lvj6\" (UID: \"b83b973c-2a27-4338-b700-0a98903f17da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"
Nov 02 13:33:41.237332 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.237210    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83b973c-2a27-4338-b700-0a98903f17da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6lvj6\" (UID: \"b83b973c-2a27-4338-b700-0a98903f17da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"
Nov 02 13:33:41.237766 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.237741    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83b973c-2a27-4338-b700-0a98903f17da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6lvj6\" (UID: \"b83b973c-2a27-4338-b700-0a98903f17da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"
Nov 02 13:33:41.239518 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.239498    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83b973c-2a27-4338-b700-0a98903f17da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6lvj6\" (UID: \"b83b973c-2a27-4338-b700-0a98903f17da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"
Nov 02 13:33:41.299534 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.299478    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlgn5\" (UniqueName: \"kubernetes.io/projected/b83b973c-2a27-4338-b700-0a98903f17da-kube-api-access-rlgn5\") pod \"maas-api-revoke-sub-ae594979-6lvj6\" (UID: \"b83b973c-2a27-4338-b700-0a98903f17da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"
Nov 02 13:33:41.354526 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.354500    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"
Nov 02 13:33:41.731678 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:41.731650    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"]
Nov 02 13:33:41.733452 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:33:41.733426    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83b973c_2a27_4338_b700_0a98903f17da.slice/crio-b0203054e7818fe9ad8092e0fe1122f4c358ad5da440dd2d9f70ec601b63ff34 WatchSource:0}: Error finding container b0203054e7818fe9ad8092e0fe1122f4c358ad5da440dd2d9f70ec601b63ff34: Status 404 returned error can't find the container with id b0203054e7818fe9ad8092e0fe1122f4c358ad5da440dd2d9f70ec601b63ff34
Nov 02 13:33:42.607236 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:42.607201    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6" event={"ID":"b83b973c-2a27-4338-b700-0a98903f17da","Type":"ContainerStarted","Data":"f46cd496b6ad609f6e67cdee143ba6e277b4a066920bc4b3d09525fc7eabf51b"}
Nov 02 13:33:42.607643 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:42.607248    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6" event={"ID":"b83b973c-2a27-4338-b700-0a98903f17da","Type":"ContainerStarted","Data":"b0203054e7818fe9ad8092e0fe1122f4c358ad5da440dd2d9f70ec601b63ff34"}
Nov 02 13:33:42.730096 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:42.730043    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6" podStartSLOduration=2.7300272789999998 podStartE2EDuration="2.730027279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:33:42.726386613 +0000 UTC m=+5851.290095021" watchObservedRunningTime="2026-11-02 13:33:42.730027279 +0000 UTC m=+5851.293735689"
Nov 02 13:33:43.613938 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:43.613897    4464 generic.go:356] "Generic (PLEG): container finished" podID="b83b973c-2a27-4338-b700-0a98903f17da" containerID="f46cd496b6ad609f6e67cdee143ba6e277b4a066920bc4b3d09525fc7eabf51b" exitCode=0
Nov 02 13:33:43.614344 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:43.613946    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6" event={"ID":"b83b973c-2a27-4338-b700-0a98903f17da","Type":"ContainerDied","Data":"f46cd496b6ad609f6e67cdee143ba6e277b4a066920bc4b3d09525fc7eabf51b"}
Nov 02 13:33:44.747822 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:44.747791    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"
Nov 02 13:33:44.871109 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:44.871040    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b83b973c-2a27-4338-b700-0a98903f17da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83b973c-2a27-4338-b700-0a98903f17da-maas-api-xks-ca\") pod \"b83b973c-2a27-4338-b700-0a98903f17da\" (UID: \"b83b973c-2a27-4338-b700-0a98903f17da\") "
Nov 02 13:33:44.871245 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:44.871123    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b83b973c-2a27-4338-b700-0a98903f17da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83b973c-2a27-4338-b700-0a98903f17da-maas-api-openshift-ca\") pod \"b83b973c-2a27-4338-b700-0a98903f17da\" (UID: \"b83b973c-2a27-4338-b700-0a98903f17da\") "
Nov 02 13:33:44.871245 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:44.871181    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b83b973c-2a27-4338-b700-0a98903f17da-kube-api-access-rlgn5\" (UniqueName: \"kubernetes.io/projected/b83b973c-2a27-4338-b700-0a98903f17da-kube-api-access-rlgn5\") pod \"b83b973c-2a27-4338-b700-0a98903f17da\" (UID: \"b83b973c-2a27-4338-b700-0a98903f17da\") "
Nov 02 13:33:44.871526 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:44.871501    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83b973c-2a27-4338-b700-0a98903f17da-maas-api-openshift-ca" pod "b83b973c-2a27-4338-b700-0a98903f17da" (UID: "b83b973c-2a27-4338-b700-0a98903f17da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:33:44.873531 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:44.873501    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83b973c-2a27-4338-b700-0a98903f17da-kube-api-access-rlgn5" pod "b83b973c-2a27-4338-b700-0a98903f17da" (UID: "b83b973c-2a27-4338-b700-0a98903f17da"). InnerVolumeSpecName "kube-api-access-rlgn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:44.873639 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:44.873576    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83b973c-2a27-4338-b700-0a98903f17da-maas-api-xks-ca" pod "b83b973c-2a27-4338-b700-0a98903f17da" (UID: "b83b973c-2a27-4338-b700-0a98903f17da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:33:44.972457 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:44.972430    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83b973c-2a27-4338-b700-0a98903f17da-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:33:44.972457 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:44.972456    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlgn5\" (UniqueName: \"kubernetes.io/projected/b83b973c-2a27-4338-b700-0a98903f17da-kube-api-access-rlgn5\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:33:44.972627 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:44.972466    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83b973c-2a27-4338-b700-0a98903f17da-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:33:45.624029 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:45.623997    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"
Nov 02 13:33:45.624163 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:45.623999    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6" event={"ID":"b83b973c-2a27-4338-b700-0a98903f17da","Type":"ContainerDied","Data":"b0203054e7818fe9ad8092e0fe1122f4c358ad5da440dd2d9f70ec601b63ff34"}
Nov 02 13:33:45.624163 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:45.624115    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0203054e7818fe9ad8092e0fe1122f4c358ad5da440dd2d9f70ec601b63ff34"
Nov 02 13:33:53.909907 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:33:53.909872    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:33:58.291388 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:33:58.291350    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:34:03.887228 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:03.887193    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:34:04.044332 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.044299    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"]
Nov 02 13:34:04.044806 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.044788    4464 cpu_manager.go:414] "containerMap: removing container" podUID="b83b973c-2a27-4338-b700-0a98903f17da" containerName="revoke-keys"
Nov 02 13:34:04.044903 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.044808    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83b973c-2a27-4338-b700-0a98903f17da" containerName="revoke-keys"
Nov 02 13:34:04.044963 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.044935    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83b973c-2a27-4338-b700-0a98903f17da" containerName="revoke-keys"
Nov 02 13:34:04.047990 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.047957    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"
Nov 02 13:34:04.076012 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.075990    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b64c2\" (UniqueName: \"kubernetes.io/projected/3a86890b-8c1c-45c1-93e3-fb46790028ae-kube-api-access-b64c2\") pod \"maas-api-revoke-sub-a4aef7d7-4c2hr\" (UID: \"3a86890b-8c1c-45c1-93e3-fb46790028ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"
Nov 02 13:34:04.076122 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.076095    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a86890b-8c1c-45c1-93e3-fb46790028ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4c2hr\" (UID: \"3a86890b-8c1c-45c1-93e3-fb46790028ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"
Nov 02 13:34:04.076175 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.076153    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a86890b-8c1c-45c1-93e3-fb46790028ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4c2hr\" (UID: \"3a86890b-8c1c-45c1-93e3-fb46790028ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"
Nov 02 13:34:04.103372 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.103106    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:34:04.103372 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.103186    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:34:04.153832 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.153762    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:34:04.153948 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.153931    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:34:04.156192 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.156171    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"]
Nov 02 13:34:04.177456 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.177435    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b64c2\" (UniqueName: \"kubernetes.io/projected/3a86890b-8c1c-45c1-93e3-fb46790028ae-kube-api-access-b64c2\") pod \"maas-api-revoke-sub-a4aef7d7-4c2hr\" (UID: \"3a86890b-8c1c-45c1-93e3-fb46790028ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"
Nov 02 13:34:04.177574 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.177497    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a86890b-8c1c-45c1-93e3-fb46790028ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4c2hr\" (UID: \"3a86890b-8c1c-45c1-93e3-fb46790028ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"
Nov 02 13:34:04.177574 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.177557    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a86890b-8c1c-45c1-93e3-fb46790028ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4c2hr\" (UID: \"3a86890b-8c1c-45c1-93e3-fb46790028ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"
Nov 02 13:34:04.178091 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.178074    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a86890b-8c1c-45c1-93e3-fb46790028ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4c2hr\" (UID: \"3a86890b-8c1c-45c1-93e3-fb46790028ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"
Nov 02 13:34:04.179782 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.179765    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a86890b-8c1c-45c1-93e3-fb46790028ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-4c2hr\" (UID: \"3a86890b-8c1c-45c1-93e3-fb46790028ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"
Nov 02 13:34:04.239244 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.239224    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b64c2\" (UniqueName: \"kubernetes.io/projected/3a86890b-8c1c-45c1-93e3-fb46790028ae-kube-api-access-b64c2\") pod \"maas-api-revoke-sub-a4aef7d7-4c2hr\" (UID: \"3a86890b-8c1c-45c1-93e3-fb46790028ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"
Nov 02 13:34:04.360456 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.360435    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"
Nov 02 13:34:04.734155 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.734127    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"]
Nov 02 13:34:04.735571 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:34:04.735544    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a86890b_8c1c_45c1_93e3_fb46790028ae.slice/crio-add456c6cbed7f96cc99b51a9b9c5f9f2aca9d53e79a99d58278900a2896600b WatchSource:0}: Error finding container add456c6cbed7f96cc99b51a9b9c5f9f2aca9d53e79a99d58278900a2896600b: Status 404 returned error can't find the container with id add456c6cbed7f96cc99b51a9b9c5f9f2aca9d53e79a99d58278900a2896600b
Nov 02 13:34:04.751035 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:04.751006    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr" event={"ID":"3a86890b-8c1c-45c1-93e3-fb46790028ae","Type":"ContainerStarted","Data":"add456c6cbed7f96cc99b51a9b9c5f9f2aca9d53e79a99d58278900a2896600b"}
Nov 02 13:34:05.757495 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:05.757467    4464 generic.go:356] "Generic (PLEG): container finished" podID="3a86890b-8c1c-45c1-93e3-fb46790028ae" containerID="d294fc1517c7056f10ad547ee562a7b701edba959cf0b243821d981dc619a87b" exitCode=0
Nov 02 13:34:05.757882 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:05.757559    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr" event={"ID":"3a86890b-8c1c-45c1-93e3-fb46790028ae","Type":"ContainerDied","Data":"d294fc1517c7056f10ad547ee562a7b701edba959cf0b243821d981dc619a87b"}
Nov 02 13:34:06.892822 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:06.892796    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"
Nov 02 13:34:07.003671 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:07.003638    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a86890b-8c1c-45c1-93e3-fb46790028ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a86890b-8c1c-45c1-93e3-fb46790028ae-maas-api-xks-ca\") pod \"3a86890b-8c1c-45c1-93e3-fb46790028ae\" (UID: \"3a86890b-8c1c-45c1-93e3-fb46790028ae\") "
Nov 02 13:34:07.003821 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:07.003752    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a86890b-8c1c-45c1-93e3-fb46790028ae-kube-api-access-b64c2\" (UniqueName: \"kubernetes.io/projected/3a86890b-8c1c-45c1-93e3-fb46790028ae-kube-api-access-b64c2\") pod \"3a86890b-8c1c-45c1-93e3-fb46790028ae\" (UID: \"3a86890b-8c1c-45c1-93e3-fb46790028ae\") "
Nov 02 13:34:07.003821 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:07.003805    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a86890b-8c1c-45c1-93e3-fb46790028ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a86890b-8c1c-45c1-93e3-fb46790028ae-maas-api-openshift-ca\") pod \"3a86890b-8c1c-45c1-93e3-fb46790028ae\" (UID: \"3a86890b-8c1c-45c1-93e3-fb46790028ae\") "
Nov 02 13:34:07.004191 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:07.004170    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a86890b-8c1c-45c1-93e3-fb46790028ae-maas-api-openshift-ca" pod "3a86890b-8c1c-45c1-93e3-fb46790028ae" (UID: "3a86890b-8c1c-45c1-93e3-fb46790028ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:07.005690 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:07.005670    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a86890b-8c1c-45c1-93e3-fb46790028ae-kube-api-access-b64c2" pod "3a86890b-8c1c-45c1-93e3-fb46790028ae" (UID: "3a86890b-8c1c-45c1-93e3-fb46790028ae"). InnerVolumeSpecName "kube-api-access-b64c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:07.005690 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:07.005679    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a86890b-8c1c-45c1-93e3-fb46790028ae-maas-api-xks-ca" pod "3a86890b-8c1c-45c1-93e3-fb46790028ae" (UID: "3a86890b-8c1c-45c1-93e3-fb46790028ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:07.104897 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:07.104833    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a86890b-8c1c-45c1-93e3-fb46790028ae-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:34:07.104897 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:07.104857    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b64c2\" (UniqueName: \"kubernetes.io/projected/3a86890b-8c1c-45c1-93e3-fb46790028ae-kube-api-access-b64c2\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:34:07.104897 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:07.104867    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a86890b-8c1c-45c1-93e3-fb46790028ae-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:34:07.768611 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:07.768575    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr" event={"ID":"3a86890b-8c1c-45c1-93e3-fb46790028ae","Type":"ContainerDied","Data":"add456c6cbed7f96cc99b51a9b9c5f9f2aca9d53e79a99d58278900a2896600b"}
Nov 02 13:34:07.768782 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:07.768631    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="add456c6cbed7f96cc99b51a9b9c5f9f2aca9d53e79a99d58278900a2896600b"
Nov 02 13:34:07.768782 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:07.768589    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"
Nov 02 13:34:12.396124 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:12.396095    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:34:15.031069 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:15.031033    4464 scope.go:122] "RemoveContainer" containerID="f28d01210a2b45c7cf435241ac4cf6438f58c69ecb8906e12a877a763fd6fae6"
Nov 02 13:34:22.477068 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:22.477019    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:34:25.113169 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.091941    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"]
Nov 02 13:34:25.114509 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.114480    4464 cpu_manager.go:414] "containerMap: removing container" podUID="3a86890b-8c1c-45c1-93e3-fb46790028ae" containerName="revoke-keys"
Nov 02 13:34:25.114693 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.114677    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a86890b-8c1c-45c1-93e3-fb46790028ae" containerName="revoke-keys"
Nov 02 13:34:25.114940 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.114925    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a86890b-8c1c-45c1-93e3-fb46790028ae" containerName="revoke-keys"
Nov 02 13:34:25.129473 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.129436    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"
Nov 02 13:34:25.157008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.156954    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"]
Nov 02 13:34:25.218541 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.218508    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:34:25.218998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.218935    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:34:25.219407 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.219391    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:34:25.232710 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.232664    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3418026-660d-4af2-a0ea-5aec83c8f0bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-bbrcr\" (UID: \"a3418026-660d-4af2-a0ea-5aec83c8f0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"
Nov 02 13:34:25.232998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.232955    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3418026-660d-4af2-a0ea-5aec83c8f0bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-bbrcr\" (UID: \"a3418026-660d-4af2-a0ea-5aec83c8f0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"
Nov 02 13:34:25.233815 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.233790    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkxf4\" (UniqueName: \"kubernetes.io/projected/a3418026-660d-4af2-a0ea-5aec83c8f0bc-kube-api-access-fkxf4\") pod \"maas-api-revoke-sub-79805ca9-bbrcr\" (UID: \"a3418026-660d-4af2-a0ea-5aec83c8f0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"
Nov 02 13:34:25.315998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.315110    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:34:25.337998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.335187    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3418026-660d-4af2-a0ea-5aec83c8f0bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-bbrcr\" (UID: \"a3418026-660d-4af2-a0ea-5aec83c8f0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"
Nov 02 13:34:25.337998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.335298    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkxf4\" (UniqueName: \"kubernetes.io/projected/a3418026-660d-4af2-a0ea-5aec83c8f0bc-kube-api-access-fkxf4\") pod \"maas-api-revoke-sub-79805ca9-bbrcr\" (UID: \"a3418026-660d-4af2-a0ea-5aec83c8f0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"
Nov 02 13:34:25.337998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.335395    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3418026-660d-4af2-a0ea-5aec83c8f0bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-bbrcr\" (UID: \"a3418026-660d-4af2-a0ea-5aec83c8f0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"
Nov 02 13:34:25.337998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.336465    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3418026-660d-4af2-a0ea-5aec83c8f0bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-bbrcr\" (UID: \"a3418026-660d-4af2-a0ea-5aec83c8f0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"
Nov 02 13:34:25.342770 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.342719    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3418026-660d-4af2-a0ea-5aec83c8f0bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-bbrcr\" (UID: \"a3418026-660d-4af2-a0ea-5aec83c8f0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"
Nov 02 13:34:25.413701 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.413621    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkxf4\" (UniqueName: \"kubernetes.io/projected/a3418026-660d-4af2-a0ea-5aec83c8f0bc-kube-api-access-fkxf4\") pod \"maas-api-revoke-sub-79805ca9-bbrcr\" (UID: \"a3418026-660d-4af2-a0ea-5aec83c8f0bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"
Nov 02 13:34:25.458681 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.458653    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"
Nov 02 13:34:25.993916 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:25.993873    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"]
Nov 02 13:34:26.126553 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:26.126077    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gt29z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:34:26.126553 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:26.126155    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podUID="bf09d973-3e5b-4daa-aa7f-cf839b61d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:34:26.903155 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:26.903111    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr" event={"ID":"a3418026-660d-4af2-a0ea-5aec83c8f0bc","Type":"ContainerStarted","Data":"61496b44e0f213ed7613a6acbc80fa91a02f851ffdeb91bf92d06b80878f00be"}
Nov 02 13:34:26.903155 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:26.903162    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr" event={"ID":"a3418026-660d-4af2-a0ea-5aec83c8f0bc","Type":"ContainerStarted","Data":"909ee73d41ce8f9d1f855e8a496defbac6f4d8a00071b44a019db9d1ad030e36"}
Nov 02 13:34:27.021393 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:27.021338    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr" podStartSLOduration=2.021322933 podStartE2EDuration="2.021322933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:34:27.02050089 +0000 UTC m=+5895.584209302" watchObservedRunningTime="2026-11-02 13:34:27.021322933 +0000 UTC m=+5895.585031342"
Nov 02 13:34:27.909842 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:27.909803    4464 generic.go:356] "Generic (PLEG): container finished" podID="a3418026-660d-4af2-a0ea-5aec83c8f0bc" containerID="61496b44e0f213ed7613a6acbc80fa91a02f851ffdeb91bf92d06b80878f00be" exitCode=0
Nov 02 13:34:27.910283 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:27.909859    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr" event={"ID":"a3418026-660d-4af2-a0ea-5aec83c8f0bc","Type":"ContainerDied","Data":"61496b44e0f213ed7613a6acbc80fa91a02f851ffdeb91bf92d06b80878f00be"}
Nov 02 13:34:28.291247 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:34:28.291213    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:34:29.043958 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:29.043933    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"
Nov 02 13:34:29.071201 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:29.071177    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3418026-660d-4af2-a0ea-5aec83c8f0bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3418026-660d-4af2-a0ea-5aec83c8f0bc-maas-api-openshift-ca\") pod \"a3418026-660d-4af2-a0ea-5aec83c8f0bc\" (UID: \"a3418026-660d-4af2-a0ea-5aec83c8f0bc\") "
Nov 02 13:34:29.071299 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:29.071252    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3418026-660d-4af2-a0ea-5aec83c8f0bc-kube-api-access-fkxf4\" (UniqueName: \"kubernetes.io/projected/a3418026-660d-4af2-a0ea-5aec83c8f0bc-kube-api-access-fkxf4\") pod \"a3418026-660d-4af2-a0ea-5aec83c8f0bc\" (UID: \"a3418026-660d-4af2-a0ea-5aec83c8f0bc\") "
Nov 02 13:34:29.071337 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:29.071298    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3418026-660d-4af2-a0ea-5aec83c8f0bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3418026-660d-4af2-a0ea-5aec83c8f0bc-maas-api-xks-ca\") pod \"a3418026-660d-4af2-a0ea-5aec83c8f0bc\" (UID: \"a3418026-660d-4af2-a0ea-5aec83c8f0bc\") "
Nov 02 13:34:29.071499 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:29.071478    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3418026-660d-4af2-a0ea-5aec83c8f0bc-maas-api-openshift-ca" pod "a3418026-660d-4af2-a0ea-5aec83c8f0bc" (UID: "a3418026-660d-4af2-a0ea-5aec83c8f0bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:29.071664 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:29.071637    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3418026-660d-4af2-a0ea-5aec83c8f0bc-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:34:29.073201 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:29.073179    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3418026-660d-4af2-a0ea-5aec83c8f0bc-kube-api-access-fkxf4" pod "a3418026-660d-4af2-a0ea-5aec83c8f0bc" (UID: "a3418026-660d-4af2-a0ea-5aec83c8f0bc"). InnerVolumeSpecName "kube-api-access-fkxf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:29.073347 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:29.073328    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3418026-660d-4af2-a0ea-5aec83c8f0bc-maas-api-xks-ca" pod "a3418026-660d-4af2-a0ea-5aec83c8f0bc" (UID: "a3418026-660d-4af2-a0ea-5aec83c8f0bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:29.172676 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:29.172595    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkxf4\" (UniqueName: \"kubernetes.io/projected/a3418026-660d-4af2-a0ea-5aec83c8f0bc-kube-api-access-fkxf4\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:34:29.172676 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:29.172621    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3418026-660d-4af2-a0ea-5aec83c8f0bc-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:34:29.921037 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:29.920953    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr" event={"ID":"a3418026-660d-4af2-a0ea-5aec83c8f0bc","Type":"ContainerDied","Data":"909ee73d41ce8f9d1f855e8a496defbac6f4d8a00071b44a019db9d1ad030e36"}
Nov 02 13:34:29.921037 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:29.920995    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"
Nov 02 13:34:29.921037 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:29.921027    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="909ee73d41ce8f9d1f855e8a496defbac6f4d8a00071b44a019db9d1ad030e36"
Nov 02 13:34:31.987865 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:31.987827    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:34:41.087922 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:41.087884    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:34:42.467547 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.467515    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"]
Nov 02 13:34:42.468234 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.468211    4464 cpu_manager.go:414] "containerMap: removing container" podUID="a3418026-660d-4af2-a0ea-5aec83c8f0bc" containerName="revoke-keys"
Nov 02 13:34:42.468294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.468238    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3418026-660d-4af2-a0ea-5aec83c8f0bc" containerName="revoke-keys"
Nov 02 13:34:42.468404 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.468390    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3418026-660d-4af2-a0ea-5aec83c8f0bc" containerName="revoke-keys"
Nov 02 13:34:42.472165 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.472142    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"
Nov 02 13:34:42.532948 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.532913    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"]
Nov 02 13:34:42.576691 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.576652    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:34:42.576926 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.576660    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:34:42.577074 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.576676    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:34:42.577074 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.576706    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:34:42.602737 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.602697    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34cbe4ca-93a9-4410-9d83-9d82d7abc152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fq8fl\" (UID: \"34cbe4ca-93a9-4410-9d83-9d82d7abc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"
Nov 02 13:34:42.602890 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.602804    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34cbe4ca-93a9-4410-9d83-9d82d7abc152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fq8fl\" (UID: \"34cbe4ca-93a9-4410-9d83-9d82d7abc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"
Nov 02 13:34:42.602890 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.602851    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jdmb\" (UniqueName: \"kubernetes.io/projected/34cbe4ca-93a9-4410-9d83-9d82d7abc152-kube-api-access-5jdmb\") pod \"maas-api-revoke-sub-0b7e8edb-fq8fl\" (UID: \"34cbe4ca-93a9-4410-9d83-9d82d7abc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"
Nov 02 13:34:42.703873 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.703835    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdmb\" (UniqueName: \"kubernetes.io/projected/34cbe4ca-93a9-4410-9d83-9d82d7abc152-kube-api-access-5jdmb\") pod \"maas-api-revoke-sub-0b7e8edb-fq8fl\" (UID: \"34cbe4ca-93a9-4410-9d83-9d82d7abc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"
Nov 02 13:34:42.704245 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.704215    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34cbe4ca-93a9-4410-9d83-9d82d7abc152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fq8fl\" (UID: \"34cbe4ca-93a9-4410-9d83-9d82d7abc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"
Nov 02 13:34:42.704571 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.704542    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34cbe4ca-93a9-4410-9d83-9d82d7abc152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fq8fl\" (UID: \"34cbe4ca-93a9-4410-9d83-9d82d7abc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"
Nov 02 13:34:42.705457 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.705430    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34cbe4ca-93a9-4410-9d83-9d82d7abc152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fq8fl\" (UID: \"34cbe4ca-93a9-4410-9d83-9d82d7abc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"
Nov 02 13:34:42.707569 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.707542    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34cbe4ca-93a9-4410-9d83-9d82d7abc152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-fq8fl\" (UID: \"34cbe4ca-93a9-4410-9d83-9d82d7abc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"
Nov 02 13:34:42.768431 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.768362    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jdmb\" (UniqueName: \"kubernetes.io/projected/34cbe4ca-93a9-4410-9d83-9d82d7abc152-kube-api-access-5jdmb\") pod \"maas-api-revoke-sub-0b7e8edb-fq8fl\" (UID: \"34cbe4ca-93a9-4410-9d83-9d82d7abc152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"
Nov 02 13:34:42.800176 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:42.800137    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"
Nov 02 13:34:43.307357 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:43.298350    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"]
Nov 02 13:34:44.081008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:44.079074    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl" event={"ID":"34cbe4ca-93a9-4410-9d83-9d82d7abc152","Type":"ContainerStarted","Data":"5c708e26d80fb3ff89bf39c0310fc469e06081f6ab814b66e54ec6f4ab586c45"}
Nov 02 13:34:44.081008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:44.079145    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl" event={"ID":"34cbe4ca-93a9-4410-9d83-9d82d7abc152","Type":"ContainerStarted","Data":"db40198ed6bb89a3afa46c61b7f933e7dcc0042467914992dc8078b58ebf37b0"}
Nov 02 13:34:44.204010 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:44.203743    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl" podStartSLOduration=2.203723781 podStartE2EDuration="2.203723781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:34:44.20131521 +0000 UTC m=+5912.765023621" watchObservedRunningTime="2026-11-02 13:34:44.203723781 +0000 UTC m=+5912.767432192"
Nov 02 13:34:45.119001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:45.118718    4464 generic.go:356] "Generic (PLEG): container finished" podID="34cbe4ca-93a9-4410-9d83-9d82d7abc152" containerID="5c708e26d80fb3ff89bf39c0310fc469e06081f6ab814b66e54ec6f4ab586c45" exitCode=0
Nov 02 13:34:45.119001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:45.118800    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl" event={"ID":"34cbe4ca-93a9-4410-9d83-9d82d7abc152","Type":"ContainerDied","Data":"5c708e26d80fb3ff89bf39c0310fc469e06081f6ab814b66e54ec6f4ab586c45"}
Nov 02 13:34:46.269769 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:46.269741    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"
Nov 02 13:34:46.369651 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:46.369622    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34cbe4ca-93a9-4410-9d83-9d82d7abc152-kube-api-access-5jdmb\" (UniqueName: \"kubernetes.io/projected/34cbe4ca-93a9-4410-9d83-9d82d7abc152-kube-api-access-5jdmb\") pod \"34cbe4ca-93a9-4410-9d83-9d82d7abc152\" (UID: \"34cbe4ca-93a9-4410-9d83-9d82d7abc152\") "
Nov 02 13:34:46.369840 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:46.369683    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34cbe4ca-93a9-4410-9d83-9d82d7abc152-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34cbe4ca-93a9-4410-9d83-9d82d7abc152-maas-api-xks-ca\") pod \"34cbe4ca-93a9-4410-9d83-9d82d7abc152\" (UID: \"34cbe4ca-93a9-4410-9d83-9d82d7abc152\") "
Nov 02 13:34:46.369840 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:46.369807    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34cbe4ca-93a9-4410-9d83-9d82d7abc152-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34cbe4ca-93a9-4410-9d83-9d82d7abc152-maas-api-openshift-ca\") pod \"34cbe4ca-93a9-4410-9d83-9d82d7abc152\" (UID: \"34cbe4ca-93a9-4410-9d83-9d82d7abc152\") "
Nov 02 13:34:46.370240 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:46.370210    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34cbe4ca-93a9-4410-9d83-9d82d7abc152-maas-api-openshift-ca" pod "34cbe4ca-93a9-4410-9d83-9d82d7abc152" (UID: "34cbe4ca-93a9-4410-9d83-9d82d7abc152"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:46.371755 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:46.371730    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34cbe4ca-93a9-4410-9d83-9d82d7abc152-maas-api-xks-ca" pod "34cbe4ca-93a9-4410-9d83-9d82d7abc152" (UID: "34cbe4ca-93a9-4410-9d83-9d82d7abc152"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:46.371860 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:46.371811    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34cbe4ca-93a9-4410-9d83-9d82d7abc152-kube-api-access-5jdmb" pod "34cbe4ca-93a9-4410-9d83-9d82d7abc152" (UID: "34cbe4ca-93a9-4410-9d83-9d82d7abc152"). InnerVolumeSpecName "kube-api-access-5jdmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:46.471035 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:46.471006    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34cbe4ca-93a9-4410-9d83-9d82d7abc152-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:34:46.471035 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:46.471032    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jdmb\" (UniqueName: \"kubernetes.io/projected/34cbe4ca-93a9-4410-9d83-9d82d7abc152-kube-api-access-5jdmb\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:34:46.471035 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:46.471042    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34cbe4ca-93a9-4410-9d83-9d82d7abc152-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:34:47.130332 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:47.130299    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl" event={"ID":"34cbe4ca-93a9-4410-9d83-9d82d7abc152","Type":"ContainerDied","Data":"db40198ed6bb89a3afa46c61b7f933e7dcc0042467914992dc8078b58ebf37b0"}
Nov 02 13:34:47.130515 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:47.130351    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db40198ed6bb89a3afa46c61b7f933e7dcc0042467914992dc8078b58ebf37b0"
Nov 02 13:34:47.130515 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:47.130310    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"
Nov 02 13:34:50.806121 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:50.806086    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:34:53.700155 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.700122    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"]
Nov 02 13:34:53.700559 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.700551    4464 cpu_manager.go:414] "containerMap: removing container" podUID="34cbe4ca-93a9-4410-9d83-9d82d7abc152" containerName="revoke-keys"
Nov 02 13:34:53.700605 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.700563    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34cbe4ca-93a9-4410-9d83-9d82d7abc152" containerName="revoke-keys"
Nov 02 13:34:53.700667 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.700656    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="34cbe4ca-93a9-4410-9d83-9d82d7abc152" containerName="revoke-keys"
Nov 02 13:34:53.703934 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.703910    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"
Nov 02 13:34:53.744623 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.744597    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f859da-72c2-4166-a1ab-63d3e753b315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lc5lz\" (UID: \"00f859da-72c2-4166-a1ab-63d3e753b315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"
Nov 02 13:34:53.744781 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.744651    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnz7p\" (UniqueName: \"kubernetes.io/projected/00f859da-72c2-4166-a1ab-63d3e753b315-kube-api-access-lnz7p\") pod \"maas-api-revoke-sub-3d4156f1-lc5lz\" (UID: \"00f859da-72c2-4166-a1ab-63d3e753b315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"
Nov 02 13:34:53.744845 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.744805    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f859da-72c2-4166-a1ab-63d3e753b315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lc5lz\" (UID: \"00f859da-72c2-4166-a1ab-63d3e753b315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"
Nov 02 13:34:53.757260 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.757240    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:34:53.807949 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.807922    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:34:53.808089 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.807967    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:34:53.808089 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.807967    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:34:53.810330 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.810312    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"]
Nov 02 13:34:53.846207 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.846183    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f859da-72c2-4166-a1ab-63d3e753b315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lc5lz\" (UID: \"00f859da-72c2-4166-a1ab-63d3e753b315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"
Nov 02 13:34:53.846361 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.846226    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnz7p\" (UniqueName: \"kubernetes.io/projected/00f859da-72c2-4166-a1ab-63d3e753b315-kube-api-access-lnz7p\") pod \"maas-api-revoke-sub-3d4156f1-lc5lz\" (UID: \"00f859da-72c2-4166-a1ab-63d3e753b315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"
Nov 02 13:34:53.846361 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.846285    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f859da-72c2-4166-a1ab-63d3e753b315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lc5lz\" (UID: \"00f859da-72c2-4166-a1ab-63d3e753b315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"
Nov 02 13:34:53.846959 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.846932    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f859da-72c2-4166-a1ab-63d3e753b315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lc5lz\" (UID: \"00f859da-72c2-4166-a1ab-63d3e753b315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"
Nov 02 13:34:53.848515 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.848494    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f859da-72c2-4166-a1ab-63d3e753b315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lc5lz\" (UID: \"00f859da-72c2-4166-a1ab-63d3e753b315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"
Nov 02 13:34:53.907828 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:53.907804    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnz7p\" (UniqueName: \"kubernetes.io/projected/00f859da-72c2-4166-a1ab-63d3e753b315-kube-api-access-lnz7p\") pod \"maas-api-revoke-sub-3d4156f1-lc5lz\" (UID: \"00f859da-72c2-4166-a1ab-63d3e753b315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"
Nov 02 13:34:54.016310 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:54.016283    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"
Nov 02 13:34:54.395203 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:54.387648    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"]
Nov 02 13:34:55.175207 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:55.175173    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz" event={"ID":"00f859da-72c2-4166-a1ab-63d3e753b315","Type":"ContainerStarted","Data":"d1827e2baa3e3e1449753357e690c9c1a187af484b8f57add59b4c15eb8700ce"}
Nov 02 13:34:55.175785 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:55.175224    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz" event={"ID":"00f859da-72c2-4166-a1ab-63d3e753b315","Type":"ContainerStarted","Data":"228ebbd99becf87c5c7a26bbe40b526c5c61395a272fd3e76752d53d27ab0ee8"}
Nov 02 13:34:55.294117 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:55.294068    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz" podStartSLOduration=2.2940555910000002 podStartE2EDuration="2.294055591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:34:55.291760066 +0000 UTC m=+5923.855468477" watchObservedRunningTime="2026-11-02 13:34:55.294055591 +0000 UTC m=+5923.857764001"
Nov 02 13:34:56.181793 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:56.181757    4464 generic.go:356] "Generic (PLEG): container finished" podID="00f859da-72c2-4166-a1ab-63d3e753b315" containerID="d1827e2baa3e3e1449753357e690c9c1a187af484b8f57add59b4c15eb8700ce" exitCode=0
Nov 02 13:34:56.182260 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:56.181802    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz" event={"ID":"00f859da-72c2-4166-a1ab-63d3e753b315","Type":"ContainerDied","Data":"d1827e2baa3e3e1449753357e690c9c1a187af484b8f57add59b4c15eb8700ce"}
Nov 02 13:34:57.319384 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:57.319360    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"
Nov 02 13:34:57.380512 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:57.380480    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00f859da-72c2-4166-a1ab-63d3e753b315-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f859da-72c2-4166-a1ab-63d3e753b315-maas-api-openshift-ca\") pod \"00f859da-72c2-4166-a1ab-63d3e753b315\" (UID: \"00f859da-72c2-4166-a1ab-63d3e753b315\") "
Nov 02 13:34:57.380688 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:57.380613    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f859da-72c2-4166-a1ab-63d3e753b315-kube-api-access-lnz7p\" (UniqueName: \"kubernetes.io/projected/00f859da-72c2-4166-a1ab-63d3e753b315-kube-api-access-lnz7p\") pod \"00f859da-72c2-4166-a1ab-63d3e753b315\" (UID: \"00f859da-72c2-4166-a1ab-63d3e753b315\") "
Nov 02 13:34:57.380688 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:57.380669    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00f859da-72c2-4166-a1ab-63d3e753b315-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f859da-72c2-4166-a1ab-63d3e753b315-maas-api-xks-ca\") pod \"00f859da-72c2-4166-a1ab-63d3e753b315\" (UID: \"00f859da-72c2-4166-a1ab-63d3e753b315\") "
Nov 02 13:34:57.380813 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:57.380799    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f859da-72c2-4166-a1ab-63d3e753b315-maas-api-openshift-ca" pod "00f859da-72c2-4166-a1ab-63d3e753b315" (UID: "00f859da-72c2-4166-a1ab-63d3e753b315"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:57.380997 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:57.380964    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f859da-72c2-4166-a1ab-63d3e753b315-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:34:57.382583 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:57.382566    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f859da-72c2-4166-a1ab-63d3e753b315-maas-api-xks-ca" pod "00f859da-72c2-4166-a1ab-63d3e753b315" (UID: "00f859da-72c2-4166-a1ab-63d3e753b315"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:57.382701 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:57.382684    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f859da-72c2-4166-a1ab-63d3e753b315-kube-api-access-lnz7p" pod "00f859da-72c2-4166-a1ab-63d3e753b315" (UID: "00f859da-72c2-4166-a1ab-63d3e753b315"). InnerVolumeSpecName "kube-api-access-lnz7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:57.481834 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:57.481806    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnz7p\" (UniqueName: \"kubernetes.io/projected/00f859da-72c2-4166-a1ab-63d3e753b315-kube-api-access-lnz7p\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:34:57.481834 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:57.481835    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f859da-72c2-4166-a1ab-63d3e753b315-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:34:58.193217 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:58.193185    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"
Nov 02 13:34:58.193404 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:58.193220    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz" event={"ID":"00f859da-72c2-4166-a1ab-63d3e753b315","Type":"ContainerDied","Data":"228ebbd99becf87c5c7a26bbe40b526c5c61395a272fd3e76752d53d27ab0ee8"}
Nov 02 13:34:58.193404 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:34:58.193276    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="228ebbd99becf87c5c7a26bbe40b526c5c61395a272fd3e76752d53d27ab0ee8"
Nov 02 13:34:58.291563 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:34:58.291530    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:35:08.275381 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:08.275347    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:35:11.129001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:11.128093    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gt29z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:35:11.129001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:11.128188    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podUID="bf09d973-3e5b-4daa-aa7f-cf839b61d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:35:19.988472 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:19.988431    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:35:21.869821 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:21.869789    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"]
Nov 02 13:35:21.920368 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:21.920333    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lc5lz"]
Nov 02 13:35:21.995715 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:21.995683    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"]
Nov 02 13:35:21.996122 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:21.996109    4464 cpu_manager.go:414] "containerMap: removing container" podUID="00f859da-72c2-4166-a1ab-63d3e753b315" containerName="revoke-keys"
Nov 02 13:35:21.996177 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:21.996123    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f859da-72c2-4166-a1ab-63d3e753b315" containerName="revoke-keys"
Nov 02 13:35:21.996237 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:21.996228    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f859da-72c2-4166-a1ab-63d3e753b315" containerName="revoke-keys"
Nov 02 13:35:21.999328 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:21.999313    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"
Nov 02 13:35:22.054268 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:22.054226    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:35:22.054268 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:22.054250    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:35:22.054626 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:22.054599    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:35:22.054764 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:22.054694    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:35:22.058458 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:22.058436    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"]
Nov 02 13:35:22.084283 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:22.084252    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5kwd\" (UniqueName: \"kubernetes.io/projected/d59f942c-114f-4d62-bf82-51e9336103ac-kube-api-access-n5kwd\") pod \"maas-api-revoke-sub-3d4156f1-vgz72\" (UID: \"d59f942c-114f-4d62-bf82-51e9336103ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"
Nov 02 13:35:22.084446 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:22.084424    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59f942c-114f-4d62-bf82-51e9336103ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vgz72\" (UID: \"d59f942c-114f-4d62-bf82-51e9336103ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"
Nov 02 13:35:22.084495 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:22.084470    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59f942c-114f-4d62-bf82-51e9336103ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vgz72\" (UID: \"d59f942c-114f-4d62-bf82-51e9336103ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"
Nov 02 13:35:22.185917 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:22.185835    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59f942c-114f-4d62-bf82-51e9336103ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vgz72\" (UID: \"d59f942c-114f-4d62-bf82-51e9336103ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"
Nov 02 13:35:22.185917 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:22.185885    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59f942c-114f-4d62-bf82-51e9336103ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vgz72\" (UID: \"d59f942c-114f-4d62-bf82-51e9336103ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"
Nov 02 13:35:22.186138 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:22.185932    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5kwd\" (UniqueName: \"kubernetes.io/projected/d59f942c-114f-4d62-bf82-51e9336103ac-kube-api-access-n5kwd\") pod \"maas-api-revoke-sub-3d4156f1-vgz72\" (UID: \"d59f942c-114f-4d62-bf82-51e9336103ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"
Nov 02 13:35:22.186653 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:22.186623    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59f942c-114f-4d62-bf82-51e9336103ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vgz72\" (UID: \"d59f942c-114f-4d62-bf82-51e9336103ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"
Nov 02 13:35:22.188242 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:22.188224    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59f942c-114f-4d62-bf82-51e9336103ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vgz72\" (UID: \"d59f942c-114f-4d62-bf82-51e9336103ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"
Nov 02 13:35:22.249378 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:22.249351    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5kwd\" (UniqueName: \"kubernetes.io/projected/d59f942c-114f-4d62-bf82-51e9336103ac-kube-api-access-n5kwd\") pod \"maas-api-revoke-sub-3d4156f1-vgz72\" (UID: \"d59f942c-114f-4d62-bf82-51e9336103ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"
Nov 02 13:35:22.311661 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:22.311630    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"
Nov 02 13:35:22.685048 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:22.685012    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"]
Nov 02 13:35:22.686465 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:35:22.686435    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59f942c_114f_4d62_bf82_51e9336103ac.slice/crio-4eed495c6f13aea78485baea2e6bedd6a44c4374ac0d753514f5712f6a5efa2f WatchSource:0}: Error finding container 4eed495c6f13aea78485baea2e6bedd6a44c4374ac0d753514f5712f6a5efa2f: Status 404 returned error can't find the container with id 4eed495c6f13aea78485baea2e6bedd6a44c4374ac0d753514f5712f6a5efa2f
Nov 02 13:35:23.383294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:23.383257    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72" event={"ID":"d59f942c-114f-4d62-bf82-51e9336103ac","Type":"ContainerStarted","Data":"17e41261cdb2566a4ed0c2a7fb03458afa0ff3d8f205befdca4434e62ebafaed"}
Nov 02 13:35:23.383695 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:23.383322    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72" event={"ID":"d59f942c-114f-4d62-bf82-51e9336103ac","Type":"ContainerStarted","Data":"4eed495c6f13aea78485baea2e6bedd6a44c4374ac0d753514f5712f6a5efa2f"}
Nov 02 13:35:23.503284 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:23.503240    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72" podStartSLOduration=2.503227089 podStartE2EDuration="2.503227089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:35:23.500624285 +0000 UTC m=+5952.064332694" watchObservedRunningTime="2026-11-02 13:35:23.503227089 +0000 UTC m=+5952.066935497"
Nov 02 13:35:23.580740 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:23.580707    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f859da-72c2-4166-a1ab-63d3e753b315" path="/var/lib/kubelet/pods/00f859da-72c2-4166-a1ab-63d3e753b315/volumes"
Nov 02 13:35:24.390088 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:24.390055    4464 generic.go:356] "Generic (PLEG): container finished" podID="d59f942c-114f-4d62-bf82-51e9336103ac" containerID="17e41261cdb2566a4ed0c2a7fb03458afa0ff3d8f205befdca4434e62ebafaed" exitCode=0
Nov 02 13:35:24.390501 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:24.390144    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72" event={"ID":"d59f942c-114f-4d62-bf82-51e9336103ac","Type":"ContainerDied","Data":"17e41261cdb2566a4ed0c2a7fb03458afa0ff3d8f205befdca4434e62ebafaed"}
Nov 02 13:35:25.531069 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:25.531038    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"
Nov 02 13:35:25.621813 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:25.621785    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d59f942c-114f-4d62-bf82-51e9336103ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59f942c-114f-4d62-bf82-51e9336103ac-maas-api-openshift-ca\") pod \"d59f942c-114f-4d62-bf82-51e9336103ac\" (UID: \"d59f942c-114f-4d62-bf82-51e9336103ac\") "
Nov 02 13:35:25.622016 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:25.621890    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59f942c-114f-4d62-bf82-51e9336103ac-kube-api-access-n5kwd\" (UniqueName: \"kubernetes.io/projected/d59f942c-114f-4d62-bf82-51e9336103ac-kube-api-access-n5kwd\") pod \"d59f942c-114f-4d62-bf82-51e9336103ac\" (UID: \"d59f942c-114f-4d62-bf82-51e9336103ac\") "
Nov 02 13:35:25.622016 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:25.621937    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d59f942c-114f-4d62-bf82-51e9336103ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59f942c-114f-4d62-bf82-51e9336103ac-maas-api-xks-ca\") pod \"d59f942c-114f-4d62-bf82-51e9336103ac\" (UID: \"d59f942c-114f-4d62-bf82-51e9336103ac\") "
Nov 02 13:35:25.622202 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:25.622175    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59f942c-114f-4d62-bf82-51e9336103ac-maas-api-openshift-ca" pod "d59f942c-114f-4d62-bf82-51e9336103ac" (UID: "d59f942c-114f-4d62-bf82-51e9336103ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:35:25.622374 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:25.622353    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d59f942c-114f-4d62-bf82-51e9336103ac-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:35:25.623991 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:25.623952    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59f942c-114f-4d62-bf82-51e9336103ac-kube-api-access-n5kwd" pod "d59f942c-114f-4d62-bf82-51e9336103ac" (UID: "d59f942c-114f-4d62-bf82-51e9336103ac"). InnerVolumeSpecName "kube-api-access-n5kwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:35:25.624137 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:25.624089    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59f942c-114f-4d62-bf82-51e9336103ac-maas-api-xks-ca" pod "d59f942c-114f-4d62-bf82-51e9336103ac" (UID: "d59f942c-114f-4d62-bf82-51e9336103ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:35:25.722904 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:25.722871    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5kwd\" (UniqueName: \"kubernetes.io/projected/d59f942c-114f-4d62-bf82-51e9336103ac-kube-api-access-n5kwd\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:35:25.722904 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:25.722901    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d59f942c-114f-4d62-bf82-51e9336103ac-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:35:26.401872 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:26.401832    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72" event={"ID":"d59f942c-114f-4d62-bf82-51e9336103ac","Type":"ContainerDied","Data":"4eed495c6f13aea78485baea2e6bedd6a44c4374ac0d753514f5712f6a5efa2f"}
Nov 02 13:35:26.401872 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:26.401852    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"
Nov 02 13:35:26.402159 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:26.401883    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4eed495c6f13aea78485baea2e6bedd6a44c4374ac0d753514f5712f6a5efa2f"
Nov 02 13:35:28.292298 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:35:28.292264    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:35:36.685591 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:36.685559    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:35:46.812456 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:46.812417    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:35:58.292237 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:35:58.292152    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:35:59.195945 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:35:59.195902    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:36:07.474894 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.474847    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"]
Nov 02 13:36:07.475434 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.475338    4464 cpu_manager.go:414] "containerMap: removing container" podUID="d59f942c-114f-4d62-bf82-51e9336103ac" containerName="revoke-keys"
Nov 02 13:36:07.475434 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.475352    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59f942c-114f-4d62-bf82-51e9336103ac" containerName="revoke-keys"
Nov 02 13:36:07.475564 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.475438    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59f942c-114f-4d62-bf82-51e9336103ac" containerName="revoke-keys"
Nov 02 13:36:07.478633 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.478612    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"
Nov 02 13:36:07.533116 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.533081    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:36:07.543615 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.543582    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90c23826-9769-4db4-b16b-9a4f329bc7cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w7gtd\" (UID: \"90c23826-9769-4db4-b16b-9a4f329bc7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"
Nov 02 13:36:07.543765 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.543647    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90c23826-9769-4db4-b16b-9a4f329bc7cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w7gtd\" (UID: \"90c23826-9769-4db4-b16b-9a4f329bc7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"
Nov 02 13:36:07.543814 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.543783    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpxsd\" (UniqueName: \"kubernetes.io/projected/90c23826-9769-4db4-b16b-9a4f329bc7cc-kube-api-access-lpxsd\") pod \"maas-api-revoke-sub-e2fc9c5b-w7gtd\" (UID: \"90c23826-9769-4db4-b16b-9a4f329bc7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"
Nov 02 13:36:07.583715 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.583677    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:36:07.583906 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.583775    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:36:07.583906 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.583826    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:36:07.586235 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.586212    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"]
Nov 02 13:36:07.644690 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.644654    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxsd\" (UniqueName: \"kubernetes.io/projected/90c23826-9769-4db4-b16b-9a4f329bc7cc-kube-api-access-lpxsd\") pod \"maas-api-revoke-sub-e2fc9c5b-w7gtd\" (UID: \"90c23826-9769-4db4-b16b-9a4f329bc7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"
Nov 02 13:36:07.644897 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.644758    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90c23826-9769-4db4-b16b-9a4f329bc7cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w7gtd\" (UID: \"90c23826-9769-4db4-b16b-9a4f329bc7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"
Nov 02 13:36:07.644897 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.644824    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90c23826-9769-4db4-b16b-9a4f329bc7cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w7gtd\" (UID: \"90c23826-9769-4db4-b16b-9a4f329bc7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"
Nov 02 13:36:07.645415 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.645393    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90c23826-9769-4db4-b16b-9a4f329bc7cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w7gtd\" (UID: \"90c23826-9769-4db4-b16b-9a4f329bc7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"
Nov 02 13:36:07.647467 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.647443    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90c23826-9769-4db4-b16b-9a4f329bc7cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w7gtd\" (UID: \"90c23826-9769-4db4-b16b-9a4f329bc7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"
Nov 02 13:36:07.706666 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.706638    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpxsd\" (UniqueName: \"kubernetes.io/projected/90c23826-9769-4db4-b16b-9a4f329bc7cc-kube-api-access-lpxsd\") pod \"maas-api-revoke-sub-e2fc9c5b-w7gtd\" (UID: \"90c23826-9769-4db4-b16b-9a4f329bc7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"
Nov 02 13:36:07.799834 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:07.799753    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"
Nov 02 13:36:08.171042 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:08.171009    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"]
Nov 02 13:36:08.171788 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:36:08.171763    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c23826_9769_4db4_b16b_9a4f329bc7cc.slice/crio-85ae42a2368a8e6a100a79f949d56115c05536ead58617a8b8424d64eaa1f0a5 WatchSource:0}: Error finding container 85ae42a2368a8e6a100a79f949d56115c05536ead58617a8b8424d64eaa1f0a5: Status 404 returned error can't find the container with id 85ae42a2368a8e6a100a79f949d56115c05536ead58617a8b8424d64eaa1f0a5
Nov 02 13:36:08.650743 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:08.650708    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd" event={"ID":"90c23826-9769-4db4-b16b-9a4f329bc7cc","Type":"ContainerStarted","Data":"1ad15d858f3900ea1ec630f47a3fea1175a657601ed0b03b4c576f2aa645b769"}
Nov 02 13:36:08.651146 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:08.650770    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd" event={"ID":"90c23826-9769-4db4-b16b-9a4f329bc7cc","Type":"ContainerStarted","Data":"85ae42a2368a8e6a100a79f949d56115c05536ead58617a8b8424d64eaa1f0a5"}
Nov 02 13:36:08.768915 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:08.768866    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd" podStartSLOduration=1.768852258 podStartE2EDuration="1.768852258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:36:08.768162838 +0000 UTC m=+5997.331871247" watchObservedRunningTime="2026-11-02 13:36:08.768852258 +0000 UTC m=+5997.332560666"
Nov 02 13:36:09.657401 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:09.657366    4464 generic.go:356] "Generic (PLEG): container finished" podID="90c23826-9769-4db4-b16b-9a4f329bc7cc" containerID="1ad15d858f3900ea1ec630f47a3fea1175a657601ed0b03b4c576f2aa645b769" exitCode=0
Nov 02 13:36:09.657818 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:09.657449    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd" event={"ID":"90c23826-9769-4db4-b16b-9a4f329bc7cc","Type":"ContainerDied","Data":"1ad15d858f3900ea1ec630f47a3fea1175a657601ed0b03b4c576f2aa645b769"}
Nov 02 13:36:10.794830 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:10.794802    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"
Nov 02 13:36:10.878601 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:10.878568    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c23826-9769-4db4-b16b-9a4f329bc7cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90c23826-9769-4db4-b16b-9a4f329bc7cc-maas-api-openshift-ca\") pod \"90c23826-9769-4db4-b16b-9a4f329bc7cc\" (UID: \"90c23826-9769-4db4-b16b-9a4f329bc7cc\") "
Nov 02 13:36:10.878776 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:10.878665    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90c23826-9769-4db4-b16b-9a4f329bc7cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90c23826-9769-4db4-b16b-9a4f329bc7cc-maas-api-xks-ca\") pod \"90c23826-9769-4db4-b16b-9a4f329bc7cc\" (UID: \"90c23826-9769-4db4-b16b-9a4f329bc7cc\") "
Nov 02 13:36:10.878776 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:10.878716    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90c23826-9769-4db4-b16b-9a4f329bc7cc-kube-api-access-lpxsd\" (UniqueName: \"kubernetes.io/projected/90c23826-9769-4db4-b16b-9a4f329bc7cc-kube-api-access-lpxsd\") pod \"90c23826-9769-4db4-b16b-9a4f329bc7cc\" (UID: \"90c23826-9769-4db4-b16b-9a4f329bc7cc\") "
Nov 02 13:36:10.878994 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:10.878946    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c23826-9769-4db4-b16b-9a4f329bc7cc-maas-api-openshift-ca" pod "90c23826-9769-4db4-b16b-9a4f329bc7cc" (UID: "90c23826-9769-4db4-b16b-9a4f329bc7cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:36:10.880724 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:10.880700    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c23826-9769-4db4-b16b-9a4f329bc7cc-kube-api-access-lpxsd" pod "90c23826-9769-4db4-b16b-9a4f329bc7cc" (UID: "90c23826-9769-4db4-b16b-9a4f329bc7cc"). InnerVolumeSpecName "kube-api-access-lpxsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:10.880811 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:10.880702    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c23826-9769-4db4-b16b-9a4f329bc7cc-maas-api-xks-ca" pod "90c23826-9769-4db4-b16b-9a4f329bc7cc" (UID: "90c23826-9769-4db4-b16b-9a4f329bc7cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:36:10.980116 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:10.980089    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90c23826-9769-4db4-b16b-9a4f329bc7cc-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:36:10.980116 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:10.980117    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpxsd\" (UniqueName: \"kubernetes.io/projected/90c23826-9769-4db4-b16b-9a4f329bc7cc-kube-api-access-lpxsd\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:36:10.980329 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:10.980128    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90c23826-9769-4db4-b16b-9a4f329bc7cc-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:36:11.668594 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:11.668559    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"
Nov 02 13:36:11.668774 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:11.668604    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd" event={"ID":"90c23826-9769-4db4-b16b-9a4f329bc7cc","Type":"ContainerDied","Data":"85ae42a2368a8e6a100a79f949d56115c05536ead58617a8b8424d64eaa1f0a5"}
Nov 02 13:36:11.668774 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:11.668668    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85ae42a2368a8e6a100a79f949d56115c05536ead58617a8b8424d64eaa1f0a5"
Nov 02 13:36:12.049214 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:12.049177    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"]
Nov 02 13:36:12.052191 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:12.052166    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xdhrs"]
Nov 02 13:36:12.345689 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:12.345612    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:36:12.361716 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:12.361693    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:36:12.399450 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:12.399428    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:36:12.415577 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:12.415554    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:36:13.581080 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:13.581047    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a21c63fd-6e08-467d-8460-5da00b64519a" path="/var/lib/kubelet/pods/a21c63fd-6e08-467d-8460-5da00b64519a/volumes"
Nov 02 13:36:18.279047 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.279014    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"]
Nov 02 13:36:18.279484 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.279470    4464 cpu_manager.go:414] "containerMap: removing container" podUID="90c23826-9769-4db4-b16b-9a4f329bc7cc" containerName="revoke-keys"
Nov 02 13:36:18.279530 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.279485    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c23826-9769-4db4-b16b-9a4f329bc7cc" containerName="revoke-keys"
Nov 02 13:36:18.279583 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.279574    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c23826-9769-4db4-b16b-9a4f329bc7cc" containerName="revoke-keys"
Nov 02 13:36:18.283927 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.283910    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"
Nov 02 13:36:18.382876 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.382846    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:36:18.382876 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.382853    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:36:18.388082 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.388063    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:36:18.388082 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.388070    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:36:18.390621 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.390592    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"]
Nov 02 13:36:18.450469 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.450375    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae727e9d-4854-4596-a194-cfdf6af320de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-5g5nn\" (UID: \"ae727e9d-4854-4596-a194-cfdf6af320de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"
Nov 02 13:36:18.450469 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.450446    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrm25\" (UniqueName: \"kubernetes.io/projected/ae727e9d-4854-4596-a194-cfdf6af320de-kube-api-access-lrm25\") pod \"maas-api-revoke-sub-5a29efb3-5g5nn\" (UID: \"ae727e9d-4854-4596-a194-cfdf6af320de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"
Nov 02 13:36:18.450715 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.450530    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae727e9d-4854-4596-a194-cfdf6af320de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-5g5nn\" (UID: \"ae727e9d-4854-4596-a194-cfdf6af320de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"
Nov 02 13:36:18.551424 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.551340    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae727e9d-4854-4596-a194-cfdf6af320de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-5g5nn\" (UID: \"ae727e9d-4854-4596-a194-cfdf6af320de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"
Nov 02 13:36:18.551424 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.551394    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrm25\" (UniqueName: \"kubernetes.io/projected/ae727e9d-4854-4596-a194-cfdf6af320de-kube-api-access-lrm25\") pod \"maas-api-revoke-sub-5a29efb3-5g5nn\" (UID: \"ae727e9d-4854-4596-a194-cfdf6af320de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"
Nov 02 13:36:18.551624 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.551426    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae727e9d-4854-4596-a194-cfdf6af320de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-5g5nn\" (UID: \"ae727e9d-4854-4596-a194-cfdf6af320de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"
Nov 02 13:36:18.552186 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.552166    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae727e9d-4854-4596-a194-cfdf6af320de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-5g5nn\" (UID: \"ae727e9d-4854-4596-a194-cfdf6af320de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"
Nov 02 13:36:18.553781 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.553762    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae727e9d-4854-4596-a194-cfdf6af320de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-5g5nn\" (UID: \"ae727e9d-4854-4596-a194-cfdf6af320de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"
Nov 02 13:36:18.613549 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.613517    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrm25\" (UniqueName: \"kubernetes.io/projected/ae727e9d-4854-4596-a194-cfdf6af320de-kube-api-access-lrm25\") pod \"maas-api-revoke-sub-5a29efb3-5g5nn\" (UID: \"ae727e9d-4854-4596-a194-cfdf6af320de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"
Nov 02 13:36:18.896382 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:18.896304    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"
Nov 02 13:36:19.270192 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:19.270163    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"]
Nov 02 13:36:19.271992 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:36:19.271948    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae727e9d_4854_4596_a194_cfdf6af320de.slice/crio-55092c84f9f85429ccc38fb436c77bfd3c38d6c000bbde7968769a04e17ec052 WatchSource:0}: Error finding container 55092c84f9f85429ccc38fb436c77bfd3c38d6c000bbde7968769a04e17ec052: Status 404 returned error can't find the container with id 55092c84f9f85429ccc38fb436c77bfd3c38d6c000bbde7968769a04e17ec052
Nov 02 13:36:19.712671 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:19.712627    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn" event={"ID":"ae727e9d-4854-4596-a194-cfdf6af320de","Type":"ContainerStarted","Data":"c9ad480a6a96b22e5a75a195cfbcf1fef0f622b759f9f24e129b1ea0e7d1341a"}
Nov 02 13:36:19.713097 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:19.712685    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn" event={"ID":"ae727e9d-4854-4596-a194-cfdf6af320de","Type":"ContainerStarted","Data":"55092c84f9f85429ccc38fb436c77bfd3c38d6c000bbde7968769a04e17ec052"}
Nov 02 13:36:19.832490 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:19.832394    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn" podStartSLOduration=1.83238037 podStartE2EDuration="1.83238037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:36:19.830359441 +0000 UTC m=+6008.394067851" watchObservedRunningTime="2026-11-02 13:36:19.83238037 +0000 UTC m=+6008.396088779"
Nov 02 13:36:20.315384 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:20.315351    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:36:20.721331 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:20.721291    4464 generic.go:356] "Generic (PLEG): container finished" podID="ae727e9d-4854-4596-a194-cfdf6af320de" containerID="c9ad480a6a96b22e5a75a195cfbcf1fef0f622b759f9f24e129b1ea0e7d1341a" exitCode=0
Nov 02 13:36:20.721810 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:20.721377    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn" event={"ID":"ae727e9d-4854-4596-a194-cfdf6af320de","Type":"ContainerDied","Data":"c9ad480a6a96b22e5a75a195cfbcf1fef0f622b759f9f24e129b1ea0e7d1341a"}
Nov 02 13:36:21.956006 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:21.955967    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"
Nov 02 13:36:22.009384 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:22.008501    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae727e9d-4854-4596-a194-cfdf6af320de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae727e9d-4854-4596-a194-cfdf6af320de-maas-api-openshift-ca\") pod \"ae727e9d-4854-4596-a194-cfdf6af320de\" (UID: \"ae727e9d-4854-4596-a194-cfdf6af320de\") "
Nov 02 13:36:22.009384 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:22.008566    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae727e9d-4854-4596-a194-cfdf6af320de-kube-api-access-lrm25\" (UniqueName: \"kubernetes.io/projected/ae727e9d-4854-4596-a194-cfdf6af320de-kube-api-access-lrm25\") pod \"ae727e9d-4854-4596-a194-cfdf6af320de\" (UID: \"ae727e9d-4854-4596-a194-cfdf6af320de\") "
Nov 02 13:36:22.009384 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:22.008677    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae727e9d-4854-4596-a194-cfdf6af320de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae727e9d-4854-4596-a194-cfdf6af320de-maas-api-xks-ca\") pod \"ae727e9d-4854-4596-a194-cfdf6af320de\" (UID: \"ae727e9d-4854-4596-a194-cfdf6af320de\") "
Nov 02 13:36:22.019188 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:22.019158    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae727e9d-4854-4596-a194-cfdf6af320de-maas-api-openshift-ca" pod "ae727e9d-4854-4596-a194-cfdf6af320de" (UID: "ae727e9d-4854-4596-a194-cfdf6af320de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:36:22.027000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:22.024444    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae727e9d-4854-4596-a194-cfdf6af320de-maas-api-xks-ca" pod "ae727e9d-4854-4596-a194-cfdf6af320de" (UID: "ae727e9d-4854-4596-a194-cfdf6af320de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:36:22.039488 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:22.039460    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae727e9d-4854-4596-a194-cfdf6af320de-kube-api-access-lrm25" pod "ae727e9d-4854-4596-a194-cfdf6af320de" (UID: "ae727e9d-4854-4596-a194-cfdf6af320de"). InnerVolumeSpecName "kube-api-access-lrm25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:22.121677 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:22.121634    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae727e9d-4854-4596-a194-cfdf6af320de-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:36:22.122161 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:22.122139    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae727e9d-4854-4596-a194-cfdf6af320de-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:36:22.122308 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:22.122291    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrm25\" (UniqueName: \"kubernetes.io/projected/ae727e9d-4854-4596-a194-cfdf6af320de-kube-api-access-lrm25\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:36:22.740155 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:22.740124    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"
Nov 02 13:36:22.740376 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:22.740158    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn" event={"ID":"ae727e9d-4854-4596-a194-cfdf6af320de","Type":"ContainerDied","Data":"55092c84f9f85429ccc38fb436c77bfd3c38d6c000bbde7968769a04e17ec052"}
Nov 02 13:36:22.740376 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:22.740203    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55092c84f9f85429ccc38fb436c77bfd3c38d6c000bbde7968769a04e17ec052"
Nov 02 13:36:28.291170 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:36:28.291138    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:36:31.187735 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:31.187699    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:36:37.058464 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.058432    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"]
Nov 02 13:36:37.108863 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.108839    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vgz72"]
Nov 02 13:36:37.182070 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.182040    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"]
Nov 02 13:36:37.182517 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.182496    4464 cpu_manager.go:414] "containerMap: removing container" podUID="ae727e9d-4854-4596-a194-cfdf6af320de" containerName="revoke-keys"
Nov 02 13:36:37.182517 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.182511    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae727e9d-4854-4596-a194-cfdf6af320de" containerName="revoke-keys"
Nov 02 13:36:37.182690 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.182588    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae727e9d-4854-4596-a194-cfdf6af320de" containerName="revoke-keys"
Nov 02 13:36:37.185940 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.185925    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"
Nov 02 13:36:37.240676 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.240657    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:36:37.240836 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.240656    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:36:37.240956 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.240687    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:36:37.240956 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.240755    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:36:37.243841 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.243815    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"]
Nov 02 13:36:37.280748 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.280717    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd739ad-127a-48f4-b1fa-73fc62634496-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vf4vt\" (UID: \"5dd739ad-127a-48f4-b1fa-73fc62634496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"
Nov 02 13:36:37.280864 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.280752    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bljt9\" (UniqueName: \"kubernetes.io/projected/5dd739ad-127a-48f4-b1fa-73fc62634496-kube-api-access-bljt9\") pod \"maas-api-revoke-sub-3d4156f1-vf4vt\" (UID: \"5dd739ad-127a-48f4-b1fa-73fc62634496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"
Nov 02 13:36:37.280864 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.280772    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd739ad-127a-48f4-b1fa-73fc62634496-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vf4vt\" (UID: \"5dd739ad-127a-48f4-b1fa-73fc62634496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"
Nov 02 13:36:37.381721 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.381662    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd739ad-127a-48f4-b1fa-73fc62634496-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vf4vt\" (UID: \"5dd739ad-127a-48f4-b1fa-73fc62634496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"
Nov 02 13:36:37.381721 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.381694    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bljt9\" (UniqueName: \"kubernetes.io/projected/5dd739ad-127a-48f4-b1fa-73fc62634496-kube-api-access-bljt9\") pod \"maas-api-revoke-sub-3d4156f1-vf4vt\" (UID: \"5dd739ad-127a-48f4-b1fa-73fc62634496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"
Nov 02 13:36:37.381721 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.381711    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd739ad-127a-48f4-b1fa-73fc62634496-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vf4vt\" (UID: \"5dd739ad-127a-48f4-b1fa-73fc62634496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"
Nov 02 13:36:37.382284 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.382267    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd739ad-127a-48f4-b1fa-73fc62634496-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-vf4vt\" (UID: \"5dd739ad-127a-48f4-b1fa-73fc62634496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"
Nov 02 13:36:37.383955 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.383936    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd739ad-127a-48f4-b1fa-73fc62634496-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-vf4vt\" (UID: \"5dd739ad-127a-48f4-b1fa-73fc62634496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"
Nov 02 13:36:37.443595 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.443566    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bljt9\" (UniqueName: \"kubernetes.io/projected/5dd739ad-127a-48f4-b1fa-73fc62634496-kube-api-access-bljt9\") pod \"maas-api-revoke-sub-3d4156f1-vf4vt\" (UID: \"5dd739ad-127a-48f4-b1fa-73fc62634496\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"
Nov 02 13:36:37.498073 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.498051    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"
Nov 02 13:36:37.581947 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.581912    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59f942c-114f-4d62-bf82-51e9336103ac" path="/var/lib/kubelet/pods/d59f942c-114f-4d62-bf82-51e9336103ac/volumes"
Nov 02 13:36:37.872327 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:37.872304    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"]
Nov 02 13:36:37.873722 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:36:37.873690    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd739ad_127a_48f4_b1fa_73fc62634496.slice/crio-610716165d6cafdcab5c52a59a1f28a0e4cffdffcee6618e9a7846954106f06c WatchSource:0}: Error finding container 610716165d6cafdcab5c52a59a1f28a0e4cffdffcee6618e9a7846954106f06c: Status 404 returned error can't find the container with id 610716165d6cafdcab5c52a59a1f28a0e4cffdffcee6618e9a7846954106f06c
Nov 02 13:36:38.824323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:38.824290    4464 generic.go:356] "Generic (PLEG): container finished" podID="5dd739ad-127a-48f4-b1fa-73fc62634496" containerID="582c2083abd3959b1a020862a65ce5447c24bcdc8b9489d03d861fa4616ce875" exitCode=0
Nov 02 13:36:38.824709 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:38.824350    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt" event={"ID":"5dd739ad-127a-48f4-b1fa-73fc62634496","Type":"ContainerDied","Data":"582c2083abd3959b1a020862a65ce5447c24bcdc8b9489d03d861fa4616ce875"}
Nov 02 13:36:38.824709 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:38.824414    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt" event={"ID":"5dd739ad-127a-48f4-b1fa-73fc62634496","Type":"ContainerStarted","Data":"610716165d6cafdcab5c52a59a1f28a0e4cffdffcee6618e9a7846954106f06c"}
Nov 02 13:36:39.966276 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:39.966252    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"
Nov 02 13:36:40.104826 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:40.104737    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dd739ad-127a-48f4-b1fa-73fc62634496-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd739ad-127a-48f4-b1fa-73fc62634496-maas-api-xks-ca\") pod \"5dd739ad-127a-48f4-b1fa-73fc62634496\" (UID: \"5dd739ad-127a-48f4-b1fa-73fc62634496\") "
Nov 02 13:36:40.105022 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:40.104834    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd739ad-127a-48f4-b1fa-73fc62634496-kube-api-access-bljt9\" (UniqueName: \"kubernetes.io/projected/5dd739ad-127a-48f4-b1fa-73fc62634496-kube-api-access-bljt9\") pod \"5dd739ad-127a-48f4-b1fa-73fc62634496\" (UID: \"5dd739ad-127a-48f4-b1fa-73fc62634496\") "
Nov 02 13:36:40.105022 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:40.104897    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dd739ad-127a-48f4-b1fa-73fc62634496-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd739ad-127a-48f4-b1fa-73fc62634496-maas-api-openshift-ca\") pod \"5dd739ad-127a-48f4-b1fa-73fc62634496\" (UID: \"5dd739ad-127a-48f4-b1fa-73fc62634496\") "
Nov 02 13:36:40.105316 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:40.105288    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd739ad-127a-48f4-b1fa-73fc62634496-maas-api-openshift-ca" pod "5dd739ad-127a-48f4-b1fa-73fc62634496" (UID: "5dd739ad-127a-48f4-b1fa-73fc62634496"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:36:40.107069 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:40.107030    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd739ad-127a-48f4-b1fa-73fc62634496-kube-api-access-bljt9" pod "5dd739ad-127a-48f4-b1fa-73fc62634496" (UID: "5dd739ad-127a-48f4-b1fa-73fc62634496"). InnerVolumeSpecName "kube-api-access-bljt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:40.107229 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:40.107206    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd739ad-127a-48f4-b1fa-73fc62634496-maas-api-xks-ca" pod "5dd739ad-127a-48f4-b1fa-73fc62634496" (UID: "5dd739ad-127a-48f4-b1fa-73fc62634496"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:36:40.206398 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:40.206366    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dd739ad-127a-48f4-b1fa-73fc62634496-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:36:40.206398 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:40.206394    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dd739ad-127a-48f4-b1fa-73fc62634496-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:36:40.206594 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:40.206407    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bljt9\" (UniqueName: \"kubernetes.io/projected/5dd739ad-127a-48f4-b1fa-73fc62634496-kube-api-access-bljt9\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:36:40.836953 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:40.836923    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"
Nov 02 13:36:40.837147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:40.836964    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt" event={"ID":"5dd739ad-127a-48f4-b1fa-73fc62634496","Type":"ContainerDied","Data":"610716165d6cafdcab5c52a59a1f28a0e4cffdffcee6618e9a7846954106f06c"}
Nov 02 13:36:40.837147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:40.837027    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="610716165d6cafdcab5c52a59a1f28a0e4cffdffcee6618e9a7846954106f06c"
Nov 02 13:36:45.049026 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:45.048971    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"]
Nov 02 13:36:45.099554 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:45.099527    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eeb23fc-lkgv4"]
Nov 02 13:36:45.580667 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:45.580633    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="034e47cc-bfb4-4d60-a810-0f720700a2a2" path="/var/lib/kubelet/pods/034e47cc-bfb4-4d60-a810-0f720700a2a2/volumes"
Nov 02 13:36:51.771063 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:51.771026    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:36:53.651335 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.649485    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"]
Nov 02 13:36:53.651335 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.650277    4464 cpu_manager.go:414] "containerMap: removing container" podUID="5dd739ad-127a-48f4-b1fa-73fc62634496" containerName="revoke-keys"
Nov 02 13:36:53.651335 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.650299    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd739ad-127a-48f4-b1fa-73fc62634496" containerName="revoke-keys"
Nov 02 13:36:53.651335 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.650435    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd739ad-127a-48f4-b1fa-73fc62634496" containerName="revoke-keys"
Nov 02 13:36:53.655596 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.655577    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.717465 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.717435    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-7ddf45-openshift-default-dockercfg-5hs5q\""
Nov 02 13:36:53.744707 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.744658    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-workload-certs\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.745128 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.745102    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bzdm\" (UniqueName: \"kubernetes.io/projected/dff4f721-0c05-44d9-acff-9d121302d980-kube-api-access-6bzdm\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.753998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.745317    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-istio-envoy\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.754327 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.754287    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-istio-data\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.754545 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.754510    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-workload-socket\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.754746 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.754715    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff4f721-0c05-44d9-acff-9d121302d980-istio-podinfo\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.755091 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.754970    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff4f721-0c05-44d9-acff-9d121302d980-istiod-ca-cert\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.755202 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.755118    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-credential-socket\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.755265 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.755203    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff4f721-0c05-44d9-acff-9d121302d980-istio-token\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.868108 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.868057    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"]
Nov 02 13:36:53.897019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.893656    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff4f721-0c05-44d9-acff-9d121302d980-istio-token\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.897019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.893711    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-workload-certs\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.897019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.893788    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bzdm\" (UniqueName: \"kubernetes.io/projected/dff4f721-0c05-44d9-acff-9d121302d980-kube-api-access-6bzdm\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.897019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.893828    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-istio-envoy\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.897019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.893890    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-istio-data\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.897019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.893923    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-workload-socket\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.897019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.893962    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff4f721-0c05-44d9-acff-9d121302d980-istio-podinfo\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.897019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.894021    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff4f721-0c05-44d9-acff-9d121302d980-istiod-ca-cert\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.897019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.894049    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-credential-socket\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.897019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.894394    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-credential-socket\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.897019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.894633    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-workload-socket\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.897019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.894882    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-istio-data\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.905091 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.905023    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff4f721-0c05-44d9-acff-9d121302d980-istiod-ca-cert\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.914012 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.913988    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-workload-certs\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.947393 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.947354    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-istio-envoy\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.952229 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.952201    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff4f721-0c05-44d9-acff-9d121302d980-istio-podinfo\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.974270 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.974239    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff4f721-0c05-44d9-acff-9d121302d980-istio-token\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.977377 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.977326    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bzdm\" (UniqueName: \"kubernetes.io/projected/dff4f721-0c05-44d9-acff-9d121302d980-kube-api-access-6bzdm\") pod \"e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") " pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:53.992678 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:53.992461    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:54.529857 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:54.529805    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"]
Nov 02 13:36:54.574999 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:36:54.569769    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff4f721_0c05_44d9_acff_9d121302d980.slice/crio-2539e62fd526e64354520df4142972e304dbcd305b152172baac60ab40dc9c06 WatchSource:0}: Error finding container 2539e62fd526e64354520df4142972e304dbcd305b152172baac60ab40dc9c06: Status 404 returned error can't find the container with id 2539e62fd526e64354520df4142972e304dbcd305b152172baac60ab40dc9c06
Nov 02 13:36:54.588181 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:54.586566    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:36:54.588181 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:54.586667    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:36:54.588181 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:54.586716    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:36:54.964473 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:54.964385    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml" event={"ID":"dff4f721-0c05-44d9-acff-9d121302d980","Type":"ContainerStarted","Data":"10e8b7d6b2baf96844be9ace8454ab88e08acef01d664ba0a2d19c3a83009187"}
Nov 02 13:36:54.964473 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:54.964445    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml" event={"ID":"dff4f721-0c05-44d9-acff-9d121302d980","Type":"ContainerStarted","Data":"2539e62fd526e64354520df4142972e304dbcd305b152172baac60ab40dc9c06"}
Nov 02 13:36:54.993380 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:54.993344    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:55.089509 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:55.089453    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml" podStartSLOduration=2.089438253 podStartE2EDuration="2.089438253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:36:55.086922359 +0000 UTC m=+6043.650630768" watchObservedRunningTime="2026-11-02 13:36:55.089438253 +0000 UTC m=+6043.653146662"
Nov 02 13:36:55.997864 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:55.997829    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:56.975297 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:56.975267    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:56.976293 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:36:56.976274    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:36:58.294723 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:36:58.294684    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:37:02.019503 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:02.019468    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"]
Nov 02 13:37:02.023235 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:02.023213    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"
Nov 02 13:37:02.114508 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:02.114483    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjv4k\" (UniqueName: \"kubernetes.io/projected/0f4088c2-3632-448b-8aee-a1da0c2462ae-kube-api-access-gjv4k\") pod \"payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2\" (UID: \"0f4088c2-3632-448b-8aee-a1da0c2462ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"
Nov 02 13:37:02.114652 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:02.114523    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4088c2-3632-448b-8aee-a1da0c2462ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2\" (UID: \"0f4088c2-3632-448b-8aee-a1da0c2462ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"
Nov 02 13:37:02.128350 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:02.128324    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-7ddf45\""
Nov 02 13:37:02.128350 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:02.128324    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-62e22d3b-dockercfg-cqwr7\""
Nov 02 13:37:02.130712 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:02.130691    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"]
Nov 02 13:37:02.215447 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:02.215419    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjv4k\" (UniqueName: \"kubernetes.io/projected/0f4088c2-3632-448b-8aee-a1da0c2462ae-kube-api-access-gjv4k\") pod \"payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2\" (UID: \"0f4088c2-3632-448b-8aee-a1da0c2462ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"
Nov 02 13:37:02.215617 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:02.215467    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4088c2-3632-448b-8aee-a1da0c2462ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2\" (UID: \"0f4088c2-3632-448b-8aee-a1da0c2462ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"
Nov 02 13:37:02.216206 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:02.216185    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4088c2-3632-448b-8aee-a1da0c2462ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2\" (UID: \"0f4088c2-3632-448b-8aee-a1da0c2462ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"
Nov 02 13:37:02.277514 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:02.277451    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjv4k\" (UniqueName: \"kubernetes.io/projected/0f4088c2-3632-448b-8aee-a1da0c2462ae-kube-api-access-gjv4k\") pod \"payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2\" (UID: \"0f4088c2-3632-448b-8aee-a1da0c2462ae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"
Nov 02 13:37:02.336790 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:02.336763    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"
Nov 02 13:37:02.712014 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:02.711984    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"]
Nov 02 13:37:02.712819 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:37:02.712796    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4088c2_3632_448b_8aee_a1da0c2462ae.slice/crio-e810a5c29712f588d53ee548736548796e41dd02caffd9b0e840e1e33c6108aa WatchSource:0}: Error finding container e810a5c29712f588d53ee548736548796e41dd02caffd9b0e840e1e33c6108aa: Status 404 returned error can't find the container with id e810a5c29712f588d53ee548736548796e41dd02caffd9b0e840e1e33c6108aa
Nov 02 13:37:03.007739 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:03.007707    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2" event={"ID":"0f4088c2-3632-448b-8aee-a1da0c2462ae","Type":"ContainerStarted","Data":"be15ad758981b1c1e86568d7cf84a7ceff8063b58493e81c80e203141c290f53"}
Nov 02 13:37:03.007914 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:03.007757    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2" event={"ID":"0f4088c2-3632-448b-8aee-a1da0c2462ae","Type":"ContainerStarted","Data":"e810a5c29712f588d53ee548736548796e41dd02caffd9b0e840e1e33c6108aa"}
Nov 02 13:37:03.007914 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:03.007867    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"
Nov 02 13:37:03.128113 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:03.128063    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2" podStartSLOduration=2.128050024 podStartE2EDuration="2.128050024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:37:03.125343623 +0000 UTC m=+6051.689052032" watchObservedRunningTime="2026-11-02 13:37:03.128050024 +0000 UTC m=+6051.691758435"
Nov 02 13:37:04.115932 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:04.115901    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:37:09.047815 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:09.047783    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"]
Nov 02 13:37:09.098284 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:09.098255    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9mjc9"]
Nov 02 13:37:09.581492 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:09.581461    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db2ebe5-57af-4f29-9cee-8c75db319b35" path="/var/lib/kubelet/pods/6db2ebe5-57af-4f29-9cee-8c75db319b35/volumes"
Nov 02 13:37:14.019098 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:14.019068    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"
Nov 02 13:37:15.084351 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:15.084319    4464 scope.go:122] "RemoveContainer" containerID="4f97e527cb0c0a20e8aaa4f80b197ddf1fc8b7f372b5614bf830f428dea38608"
Nov 02 13:37:16.187429 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:16.187395    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:37:20.123896 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:20.123855    4464 generic.go:356] "Generic (PLEG): container finished" podID="dff4f721-0c05-44d9-acff-9d121302d980" containerID="10e8b7d6b2baf96844be9ace8454ab88e08acef01d664ba0a2d19c3a83009187" exitCode=0
Nov 02 13:37:20.124344 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:20.123923    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml" event={"ID":"dff4f721-0c05-44d9-acff-9d121302d980","Type":"ContainerDied","Data":"10e8b7d6b2baf96844be9ace8454ab88e08acef01d664ba0a2d19c3a83009187"}
Nov 02 13:37:20.124344 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:20.124323    4464 scope.go:122] "RemoveContainer" containerID="10e8b7d6b2baf96844be9ace8454ab88e08acef01d664ba0a2d19c3a83009187"
Nov 02 13:37:20.125545 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:20.125520    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:37:20.125615 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:20.125582    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:37:20.125655 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:20.125614    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:37:21.130524 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:21.130489    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml" event={"ID":"dff4f721-0c05-44d9-acff-9d121302d980","Type":"ContainerStarted","Data":"8d078196ecccb493c60c190808da82bc053062feaf79b5730d60517b81a98bc0"}
Nov 02 13:37:21.993268 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:21.993232    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:37:21.993549 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:21.993526    4464 patch_prober.go:29] interesting pod/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 02 13:37:21.993640 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:21.993581    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml" podUID="dff4f721-0c05-44d9-acff-9d121302d980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 02 13:37:22.081000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:22.079251    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"]
Nov 02 13:37:22.102456 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:22.102432    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-9wn6g"]
Nov 02 13:37:22.993419 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:22.993380    4464 patch_prober.go:29] interesting pod/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 02 13:37:22.993832 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:22.993440    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml" podUID="dff4f721-0c05-44d9-acff-9d121302d980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 02 13:37:23.588378 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:23.588343    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b587cc3f-12fe-4d07-a51c-a14f1b20b581" path="/var/lib/kubelet/pods/b587cc3f-12fe-4d07-a51c-a14f1b20b581/volumes"
Nov 02 13:37:23.993460 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:23.993422    4464 patch_prober.go:29] interesting pod/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 02 13:37:23.993861 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:23.993478    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml" podUID="dff4f721-0c05-44d9-acff-9d121302d980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 02 13:37:24.995039 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:24.994400    4464 patch_prober.go:29] interesting pod/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 02 13:37:24.995039 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:24.994469    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml" podUID="dff4f721-0c05-44d9-acff-9d121302d980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 02 13:37:25.984369 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:25.984328    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"]
Nov 02 13:37:25.989371 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:25.989349    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:25.993179 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:25.993158    4464 patch_prober.go:29] interesting pod/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 02 13:37:25.993382 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:25.993358    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml" podUID="dff4f721-0c05-44d9-acff-9d121302d980" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 02 13:37:26.015627 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.015602    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-tls-certs\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.016059 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.015650    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flzcg\" (UniqueName: \"kubernetes.io/projected/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-kube-api-access-flzcg\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.016059 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.015837    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-dshm\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.016059 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.015897    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-model-cache\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.016059 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.015960    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-home\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.016059 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.016049    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-tmp-dir\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.024517 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.024492    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"]
Nov 02 13:37:26.072887 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.072850    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7ddf45\"/\"default-dockercfg-bf899\""
Nov 02 13:37:26.072887 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.072858    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7ddf45\"/\"openshift-service-ca.crt\""
Nov 02 13:37:26.072887 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.072883    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7ddf45\"/\"kube-root-ca.crt\""
Nov 02 13:37:26.094141 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.094109    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7ddf45\"/\"simulator-single-main-7ddf45-kserve-self-signed-certs\""
Nov 02 13:37:26.117999 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.117947    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-dshm\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.118306 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.118271    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-model-cache\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.118440 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.118350    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-home\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.118440 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.118405    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-tmp-dir\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.118551 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.118465    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-tls-certs\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.118551 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.118501    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flzcg\" (UniqueName: \"kubernetes.io/projected/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-kube-api-access-flzcg\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.120202 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.120177    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-model-cache\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.120398 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.120207    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-home\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.120765 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.120745    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-tmp-dir\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.128897 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.128842    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-tls-certs\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.132356 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.132328    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-dshm\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.180885 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.180856    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flzcg\" (UniqueName: \"kubernetes.io/projected/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-kube-api-access-flzcg\") pod \"simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") " pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.316194 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.316109    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:26.693577 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.693550    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"]
Nov 02 13:37:26.694579 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:37:26.694552    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a9587db_c1bc_40f0_a5c6_38cc18399ba5.slice/crio-36452cce50b56417dfc6fad1341a38ebc6e8ad6765455eaec74495ec81ca33dc WatchSource:0}: Error finding container 36452cce50b56417dfc6fad1341a38ebc6e8ad6765455eaec74495ec81ca33dc: Status 404 returned error can't find the container with id 36452cce50b56417dfc6fad1341a38ebc6e8ad6765455eaec74495ec81ca33dc
Nov 02 13:37:26.975734 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.975698    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:37:26.997399 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:26.997375    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:37:27.166363 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:27.166326    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp" event={"ID":"5a9587db-c1bc-40f0-a5c6-38cc18399ba5","Type":"ContainerStarted","Data":"a8705d88e67e614a643e56afb9f357dfeda7c8f098f61b5129310d51b327b24c"}
Nov 02 13:37:27.166363 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:27.166371    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp" event={"ID":"5a9587db-c1bc-40f0-a5c6-38cc18399ba5","Type":"ContainerStarted","Data":"36452cce50b56417dfc6fad1341a38ebc6e8ad6765455eaec74495ec81ca33dc"}
Nov 02 13:37:27.167588 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:27.167568    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:37:27.286391 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:27.286303    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp" podStartSLOduration=2.286291698 podStartE2EDuration="2.286291698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:37:27.285639484 +0000 UTC m=+6075.849347891" watchObservedRunningTime="2026-11-02 13:37:27.286291698 +0000 UTC m=+6075.850000107"
Nov 02 13:37:27.316788 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:27.316763    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:28.290756 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:37:28.290724    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:37:30.805385 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:30.805351    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:37:36.316653 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:36.316625    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:36.329290 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:36.329269    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:37.238077 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:37.238048    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:37:58.291590 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:37:58.291548    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:37:58.620747 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:37:58.620634    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:38:00.051778 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:00.051741    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"]
Nov 02 13:38:00.056204 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:00.056178    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lmmwr"]
Nov 02 13:38:01.580615 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:01.580580    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="366dc96f-3d76-4610-93e9-2813242060ff" path="/var/lib/kubelet/pods/366dc96f-3d76-4610-93e9-2813242060ff/volumes"
Nov 02 13:38:10.052055 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:10.052020    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"]
Nov 02 13:38:10.102363 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:10.102332    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-zlh69"]
Nov 02 13:38:11.581420 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:11.581383    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83b9779f-add3-47df-a72f-f2430fe99fd9" path="/var/lib/kubelet/pods/83b9779f-add3-47df-a72f-f2430fe99fd9/volumes"
Nov 02 13:38:12.363486 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:12.363448    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"]
Nov 02 13:38:12.367751 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:12.367729    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"
Nov 02 13:38:12.386416 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:12.386389    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phj6x\" (UniqueName: \"kubernetes.io/projected/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-kube-api-access-phj6x\") pod \"maas-api-revoke-sub-02824528-cq5bd\" (UID: \"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"
Nov 02 13:38:12.386551 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:12.386528    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02824528-cq5bd\" (UID: \"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"
Nov 02 13:38:12.386629 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:12.386578    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02824528-cq5bd\" (UID: \"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"
Nov 02 13:38:12.424628 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:12.424596    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"]
Nov 02 13:38:12.471616 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:12.471580    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:38:12.471810 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:12.471795    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:38:12.471885 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:12.471808    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:38:12.471885 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:12.471816    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:38:12.487539 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:12.487513    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02824528-cq5bd\" (UID: \"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"
Nov 02 13:38:12.487679 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:12.487559    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02824528-cq5bd\" (UID: \"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"
Nov 02 13:38:12.487745 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:12.487680    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phj6x\" (UniqueName: \"kubernetes.io/projected/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-kube-api-access-phj6x\") pod \"maas-api-revoke-sub-02824528-cq5bd\" (UID: \"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"
Nov 02 13:38:12.488181 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:12.488158    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02824528-cq5bd\" (UID: \"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"
Nov 02 13:38:12.489876 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:12.489857    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02824528-cq5bd\" (UID: \"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"
Nov 02 13:38:12.549744 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:12.549715    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phj6x\" (UniqueName: \"kubernetes.io/projected/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-kube-api-access-phj6x\") pod \"maas-api-revoke-sub-02824528-cq5bd\" (UID: \"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"
Nov 02 13:38:12.680631 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:12.680560    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"
Nov 02 13:38:13.055917 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:13.055887    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"]
Nov 02 13:38:13.056996 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:38:13.056959    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb46a194_e8dc_4f61_9d33_0a6ca0fd7d46.slice/crio-c232efe7b1c6d9fd9ebf68aabfa659dbb5c7c45a05e8280c5e361161b65d1278 WatchSource:0}: Error finding container c232efe7b1c6d9fd9ebf68aabfa659dbb5c7c45a05e8280c5e361161b65d1278: Status 404 returned error can't find the container with id c232efe7b1c6d9fd9ebf68aabfa659dbb5c7c45a05e8280c5e361161b65d1278
Nov 02 13:38:13.436567 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:13.436531    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd" event={"ID":"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46","Type":"ContainerStarted","Data":"bdc1bf9e6e3b56021b5f620429848b3b17ddf737d217196ff7946b20b5d82406"}
Nov 02 13:38:13.436756 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:13.436595    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd" event={"ID":"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46","Type":"ContainerStarted","Data":"c232efe7b1c6d9fd9ebf68aabfa659dbb5c7c45a05e8280c5e361161b65d1278"}
Nov 02 13:38:13.555243 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:13.555189    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd" podStartSLOduration=1.555173302 podStartE2EDuration="1.555173302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:38:13.554828379 +0000 UTC m=+6122.118536792" watchObservedRunningTime="2026-11-02 13:38:13.555173302 +0000 UTC m=+6122.118881711"
Nov 02 13:38:14.443249 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:14.443215    4464 generic.go:356] "Generic (PLEG): container finished" podID="eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46" containerID="bdc1bf9e6e3b56021b5f620429848b3b17ddf737d217196ff7946b20b5d82406" exitCode=0
Nov 02 13:38:14.443652 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:14.443297    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd" event={"ID":"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46","Type":"ContainerDied","Data":"bdc1bf9e6e3b56021b5f620429848b3b17ddf737d217196ff7946b20b5d82406"}
Nov 02 13:38:15.107318 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:15.107290    4464 scope.go:122] "RemoveContainer" containerID="9a11721b33a1498c233225766c9f9e0e18649f4da8fbc6a665cde19c14dc9b7a"
Nov 02 13:38:15.116120 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:15.116099    4464 scope.go:122] "RemoveContainer" containerID="d469ca7c6fb7ce61a043650d9ba0b4692af7efa64bdbc3662d2d3f08463fec5d"
Nov 02 13:38:15.596999 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:15.596958    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"
Nov 02 13:38:15.722584 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:15.722547    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-maas-api-openshift-ca\") pod \"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46\" (UID: \"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46\") "
Nov 02 13:38:15.722756 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:15.722631    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-kube-api-access-phj6x\" (UniqueName: \"kubernetes.io/projected/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-kube-api-access-phj6x\") pod \"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46\" (UID: \"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46\") "
Nov 02 13:38:15.722809 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:15.722752    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-maas-api-xks-ca\") pod \"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46\" (UID: \"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46\") "
Nov 02 13:38:15.722861 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:15.722819    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-maas-api-openshift-ca" pod "eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46" (UID: "eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:38:15.723190 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:15.723165    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:38:15.724769 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:15.724746    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-kube-api-access-phj6x" pod "eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46" (UID: "eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46"). InnerVolumeSpecName "kube-api-access-phj6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:38:15.724909 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:15.724782    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-maas-api-xks-ca" pod "eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46" (UID: "eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:38:15.824658 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:15.824634    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phj6x\" (UniqueName: \"kubernetes.io/projected/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-kube-api-access-phj6x\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:38:15.824658 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:15.824657    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:38:16.463992 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:16.463952    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"
Nov 02 13:38:16.464147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:16.463991    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd" event={"ID":"eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46","Type":"ContainerDied","Data":"c232efe7b1c6d9fd9ebf68aabfa659dbb5c7c45a05e8280c5e361161b65d1278"}
Nov 02 13:38:16.464147 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:16.464041    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c232efe7b1c6d9fd9ebf68aabfa659dbb5c7c45a05e8280c5e361161b65d1278"
Nov 02 13:38:27.213047 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:27.213010    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:38:28.292084 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:38:28.292049    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:38:29.072564 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:29.072527    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"]
Nov 02 13:38:31.209303 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:31.209268    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"]
Nov 02 13:38:36.209900 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:36.209853    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2" podUID="0f4088c2-3632-448b-8aee-a1da0c2462ae" containerName="payload-pre-processing" containerID="cri-o://be15ad758981b1c1e86568d7cf84a7ceff8063b58493e81c80e203141c290f53" gracePeriod=25
Nov 02 13:38:38.594333 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:38.594296    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:38:44.019029 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.018968    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2" podUID="0f4088c2-3632-448b-8aee-a1da0c2462ae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:38:44.092545 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.092506    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp" podUID="5a9587db-c1bc-40f0-a5c6-38cc18399ba5" containerName="main" containerID="cri-o://a8705d88e67e614a643e56afb9f357dfeda7c8f098f61b5129310d51b327b24c" gracePeriod=45
Nov 02 13:38:44.228037 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.228004    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp" podUID="5a9587db-c1bc-40f0-a5c6-38cc18399ba5" containerName="main" probeResult="failure" output="Get \"https://10.133.1.67:8000/ready\": dial tcp 10.133.1.67:8000: connect: connection refused"
Nov 02 13:38:44.350823 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.350799    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:38:44.517589 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.517551    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-dshm\") pod \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") "
Nov 02 13:38:44.517783 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.517598    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-tls-certs\") pod \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") "
Nov 02 13:38:44.517783 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.517675    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-model-cache\") pod \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") "
Nov 02 13:38:44.517783 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.517702    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-kube-api-access-flzcg\" (UniqueName: \"kubernetes.io/projected/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-kube-api-access-flzcg\") pod \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") "
Nov 02 13:38:44.517783 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.517762    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-tmp-dir\") pod \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") "
Nov 02 13:38:44.518069 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.517811    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-home\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-home\") pod \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\" (UID: \"5a9587db-c1bc-40f0-a5c6-38cc18399ba5\") "
Nov 02 13:38:44.518069 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.517984    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-model-cache" pod "5a9587db-c1bc-40f0-a5c6-38cc18399ba5" (UID: "5a9587db-c1bc-40f0-a5c6-38cc18399ba5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:38:44.518208 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.518062    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-tmp-dir" pod "5a9587db-c1bc-40f0-a5c6-38cc18399ba5" (UID: "5a9587db-c1bc-40f0-a5c6-38cc18399ba5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:38:44.518208 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.518196    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-home" pod "5a9587db-c1bc-40f0-a5c6-38cc18399ba5" (UID: "5a9587db-c1bc-40f0-a5c6-38cc18399ba5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:38:44.518341 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.518293    4464 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-model-cache\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:38:44.518405 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.518340    4464 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-tmp-dir\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:38:44.518405 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.518355    4464 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-home\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:38:44.519844 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.519805    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-kube-api-access-flzcg" pod "5a9587db-c1bc-40f0-a5c6-38cc18399ba5" (UID: "5a9587db-c1bc-40f0-a5c6-38cc18399ba5"). InnerVolumeSpecName "kube-api-access-flzcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:38:44.519844 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.519813    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-dshm" pod "5a9587db-c1bc-40f0-a5c6-38cc18399ba5" (UID: "5a9587db-c1bc-40f0-a5c6-38cc18399ba5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:38:44.520084 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.519886    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-tls-certs" pod "5a9587db-c1bc-40f0-a5c6-38cc18399ba5" (UID: "5a9587db-c1bc-40f0-a5c6-38cc18399ba5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:38:44.619002 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.618904    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flzcg\" (UniqueName: \"kubernetes.io/projected/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-kube-api-access-flzcg\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:38:44.619002 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.618929    4464 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-dshm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:38:44.619002 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.618939    4464 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9587db-c1bc-40f0-a5c6-38cc18399ba5-tls-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:38:44.619854 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.619830    4464 generic.go:356] "Generic (PLEG): container finished" podID="5a9587db-c1bc-40f0-a5c6-38cc18399ba5" containerID="a8705d88e67e614a643e56afb9f357dfeda7c8f098f61b5129310d51b327b24c" exitCode=0
Nov 02 13:38:44.619945 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.619877    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"
Nov 02 13:38:44.619945 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.619914    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp" event={"ID":"5a9587db-c1bc-40f0-a5c6-38cc18399ba5","Type":"ContainerDied","Data":"a8705d88e67e614a643e56afb9f357dfeda7c8f098f61b5129310d51b327b24c"}
Nov 02 13:38:44.620052 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.619957    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp" event={"ID":"5a9587db-c1bc-40f0-a5c6-38cc18399ba5","Type":"ContainerDied","Data":"36452cce50b56417dfc6fad1341a38ebc6e8ad6765455eaec74495ec81ca33dc"}
Nov 02 13:38:44.620052 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.620004    4464 scope.go:122] "RemoveContainer" containerID="a8705d88e67e614a643e56afb9f357dfeda7c8f098f61b5129310d51b327b24c"
Nov 02 13:38:44.629680 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.629665    4464 scope.go:122] "RemoveContainer" containerID="a8705d88e67e614a643e56afb9f357dfeda7c8f098f61b5129310d51b327b24c"
Nov 02 13:38:44.629924 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:38:44.629907    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8705d88e67e614a643e56afb9f357dfeda7c8f098f61b5129310d51b327b24c\": container with ID starting with a8705d88e67e614a643e56afb9f357dfeda7c8f098f61b5129310d51b327b24c not found: ID does not exist" containerID="a8705d88e67e614a643e56afb9f357dfeda7c8f098f61b5129310d51b327b24c"
Nov 02 13:38:44.630026 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.629932    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8705d88e67e614a643e56afb9f357dfeda7c8f098f61b5129310d51b327b24c"} err="failed to get container status \"a8705d88e67e614a643e56afb9f357dfeda7c8f098f61b5129310d51b327b24c\": rpc error: code = NotFound desc = could not find container \"a8705d88e67e614a643e56afb9f357dfeda7c8f098f61b5129310d51b327b24c\": container with ID starting with a8705d88e67e614a643e56afb9f357dfeda7c8f098f61b5129310d51b327b24c not found: ID does not exist"
Nov 02 13:38:44.786807 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.786781    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"]
Nov 02 13:38:44.790684 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:44.790657    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-7ddf45/simulator-single-main-7ddf45-kserve-6b4dd45dfb-7qfpp"]
Nov 02 13:38:45.048494 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:45.048465    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"]
Nov 02 13:38:45.051835 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:45.051811    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6lvj6"]
Nov 02 13:38:45.582358 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:45.582328    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9587db-c1bc-40f0-a5c6-38cc18399ba5" path="/var/lib/kubelet/pods/5a9587db-c1bc-40f0-a5c6-38cc18399ba5/volumes"
Nov 02 13:38:45.582744 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:45.582731    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b83b973c-2a27-4338-b700-0a98903f17da" path="/var/lib/kubelet/pods/b83b973c-2a27-4338-b700-0a98903f17da/volumes"
Nov 02 13:38:54.017613 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:54.017580    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2" podUID="0f4088c2-3632-448b-8aee-a1da0c2462ae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:38:56.457163 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:56.457134    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"
Nov 02 13:38:56.530221 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:56.530194    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4088c2-3632-448b-8aee-a1da0c2462ae-kube-api-access-gjv4k\" (UniqueName: \"kubernetes.io/projected/0f4088c2-3632-448b-8aee-a1da0c2462ae-kube-api-access-gjv4k\") pod \"0f4088c2-3632-448b-8aee-a1da0c2462ae\" (UID: \"0f4088c2-3632-448b-8aee-a1da0c2462ae\") "
Nov 02 13:38:56.530403 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:56.530258    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f4088c2-3632-448b-8aee-a1da0c2462ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4088c2-3632-448b-8aee-a1da0c2462ae-plugins-config-volume\") pod \"0f4088c2-3632-448b-8aee-a1da0c2462ae\" (UID: \"0f4088c2-3632-448b-8aee-a1da0c2462ae\") "
Nov 02 13:38:56.530669 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:56.530644    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4088c2-3632-448b-8aee-a1da0c2462ae-plugins-config-volume" pod "0f4088c2-3632-448b-8aee-a1da0c2462ae" (UID: "0f4088c2-3632-448b-8aee-a1da0c2462ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:38:56.532208 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:56.532186    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4088c2-3632-448b-8aee-a1da0c2462ae-kube-api-access-gjv4k" pod "0f4088c2-3632-448b-8aee-a1da0c2462ae" (UID: "0f4088c2-3632-448b-8aee-a1da0c2462ae"). InnerVolumeSpecName "kube-api-access-gjv4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:38:56.632122 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:56.632088    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjv4k\" (UniqueName: \"kubernetes.io/projected/0f4088c2-3632-448b-8aee-a1da0c2462ae-kube-api-access-gjv4k\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:38:56.632122 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:56.632124    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f4088c2-3632-448b-8aee-a1da0c2462ae-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:38:56.686323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:56.686291    4464 generic.go:356] "Generic (PLEG): container finished" podID="0f4088c2-3632-448b-8aee-a1da0c2462ae" containerID="be15ad758981b1c1e86568d7cf84a7ceff8063b58493e81c80e203141c290f53" exitCode=0
Nov 02 13:38:56.686497 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:56.686339    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"
Nov 02 13:38:56.686497 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:56.686369    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2" event={"ID":"0f4088c2-3632-448b-8aee-a1da0c2462ae","Type":"ContainerDied","Data":"be15ad758981b1c1e86568d7cf84a7ceff8063b58493e81c80e203141c290f53"}
Nov 02 13:38:56.686497 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:56.686430    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2" event={"ID":"0f4088c2-3632-448b-8aee-a1da0c2462ae","Type":"ContainerDied","Data":"e810a5c29712f588d53ee548736548796e41dd02caffd9b0e840e1e33c6108aa"}
Nov 02 13:38:56.686497 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:56.686452    4464 scope.go:122] "RemoveContainer" containerID="be15ad758981b1c1e86568d7cf84a7ceff8063b58493e81c80e203141c290f53"
Nov 02 13:38:56.695949 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:56.695936    4464 scope.go:122] "RemoveContainer" containerID="be15ad758981b1c1e86568d7cf84a7ceff8063b58493e81c80e203141c290f53"
Nov 02 13:38:56.696222 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:38:56.696204    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be15ad758981b1c1e86568d7cf84a7ceff8063b58493e81c80e203141c290f53\": container with ID starting with be15ad758981b1c1e86568d7cf84a7ceff8063b58493e81c80e203141c290f53 not found: ID does not exist" containerID="be15ad758981b1c1e86568d7cf84a7ceff8063b58493e81c80e203141c290f53"
Nov 02 13:38:56.696275 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:56.696229    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be15ad758981b1c1e86568d7cf84a7ceff8063b58493e81c80e203141c290f53"} err="failed to get container status \"be15ad758981b1c1e86568d7cf84a7ceff8063b58493e81c80e203141c290f53\": rpc error: code = NotFound desc = could not find container \"be15ad758981b1c1e86568d7cf84a7ceff8063b58493e81c80e203141c290f53\": container with ID starting with be15ad758981b1c1e86568d7cf84a7ceff8063b58493e81c80e203141c290f53 not found: ID does not exist"
Nov 02 13:38:56.854149 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:56.854064    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"]
Nov 02 13:38:56.860186 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:56.860158    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887852k2"]
Nov 02 13:38:57.582369 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:38:57.582337    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f4088c2-3632-448b-8aee-a1da0c2462ae" path="/var/lib/kubelet/pods/0f4088c2-3632-448b-8aee-a1da0c2462ae/volumes"
Nov 02 13:38:58.291897 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:38:58.291864    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:39:08.048259 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:08.048224    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"]
Nov 02 13:39:08.098423 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:08.098393    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-4c2hr"]
Nov 02 13:39:09.506690 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.506655    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"]
Nov 02 13:39:09.507143 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.507129    4464 cpu_manager.go:414] "containerMap: removing container" podUID="5a9587db-c1bc-40f0-a5c6-38cc18399ba5" containerName="main"
Nov 02 13:39:09.507202 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.507144    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9587db-c1bc-40f0-a5c6-38cc18399ba5" containerName="main"
Nov 02 13:39:09.507202 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.507165    4464 cpu_manager.go:414] "containerMap: removing container" podUID="0f4088c2-3632-448b-8aee-a1da0c2462ae" containerName="payload-pre-processing"
Nov 02 13:39:09.507202 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.507170    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f4088c2-3632-448b-8aee-a1da0c2462ae" containerName="payload-pre-processing"
Nov 02 13:39:09.507202 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.507183    4464 cpu_manager.go:414] "containerMap: removing container" podUID="eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46" containerName="revoke-keys"
Nov 02 13:39:09.507202 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.507187    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46" containerName="revoke-keys"
Nov 02 13:39:09.507363 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.507257    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9587db-c1bc-40f0-a5c6-38cc18399ba5" containerName="main"
Nov 02 13:39:09.507363 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.507268    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f4088c2-3632-448b-8aee-a1da0c2462ae" containerName="payload-pre-processing"
Nov 02 13:39:09.507363 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.507277    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46" containerName="revoke-keys"
Nov 02 13:39:09.510350 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.510334    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"
Nov 02 13:39:09.581221 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.581196    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a86890b-8c1c-45c1-93e3-fb46790028ae" path="/var/lib/kubelet/pods/3a86890b-8c1c-45c1-93e3-fb46790028ae/volumes"
Nov 02 13:39:09.615365 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.615332    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-7ddf45\""
Nov 02 13:39:09.615576 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.615421    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-62e22d3b-dockercfg-4xlkm\""
Nov 02 13:39:09.617822 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.617797    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"]
Nov 02 13:39:09.663033 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.662998    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j8jf\" (UniqueName: \"kubernetes.io/projected/a0d0ec52-b21e-484d-9eeb-04ccb55bd150-kube-api-access-8j8jf\") pod \"payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4\" (UID: \"a0d0ec52-b21e-484d-9eeb-04ccb55bd150\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"
Nov 02 13:39:09.663255 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.663231    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0d0ec52-b21e-484d-9eeb-04ccb55bd150-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4\" (UID: \"a0d0ec52-b21e-484d-9eeb-04ccb55bd150\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"
Nov 02 13:39:09.663414 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.663346    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"]
Nov 02 13:39:09.683898 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.683857    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"
Nov 02 13:39:09.728701 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.728667    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"]
Nov 02 13:39:09.764033 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.763942    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0d0ec52-b21e-484d-9eeb-04ccb55bd150-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4\" (UID: \"a0d0ec52-b21e-484d-9eeb-04ccb55bd150\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"
Nov 02 13:39:09.764033 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.764004    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j8jf\" (UniqueName: \"kubernetes.io/projected/a0d0ec52-b21e-484d-9eeb-04ccb55bd150-kube-api-access-8j8jf\") pod \"payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4\" (UID: \"a0d0ec52-b21e-484d-9eeb-04ccb55bd150\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"
Nov 02 13:39:09.764244 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.764091    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a43544-b4a0-4a97-a686-1bdf53299765-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm\" (UID: \"23a43544-b4a0-4a97-a686-1bdf53299765\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"
Nov 02 13:39:09.764244 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.764127    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjsdr\" (UniqueName: \"kubernetes.io/projected/23a43544-b4a0-4a97-a686-1bdf53299765-kube-api-access-vjsdr\") pod \"payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm\" (UID: \"23a43544-b4a0-4a97-a686-1bdf53299765\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"
Nov 02 13:39:09.764665 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.764646    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0d0ec52-b21e-484d-9eeb-04ccb55bd150-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4\" (UID: \"a0d0ec52-b21e-484d-9eeb-04ccb55bd150\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"
Nov 02 13:39:09.834910 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.834871    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j8jf\" (UniqueName: \"kubernetes.io/projected/a0d0ec52-b21e-484d-9eeb-04ccb55bd150-kube-api-access-8j8jf\") pod \"payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4\" (UID: \"a0d0ec52-b21e-484d-9eeb-04ccb55bd150\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"
Nov 02 13:39:09.864700 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.864668    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a43544-b4a0-4a97-a686-1bdf53299765-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm\" (UID: \"23a43544-b4a0-4a97-a686-1bdf53299765\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"
Nov 02 13:39:09.864877 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.864715    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjsdr\" (UniqueName: \"kubernetes.io/projected/23a43544-b4a0-4a97-a686-1bdf53299765-kube-api-access-vjsdr\") pod \"payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm\" (UID: \"23a43544-b4a0-4a97-a686-1bdf53299765\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"
Nov 02 13:39:09.865465 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.865446    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a43544-b4a0-4a97-a686-1bdf53299765-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm\" (UID: \"23a43544-b4a0-4a97-a686-1bdf53299765\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"
Nov 02 13:39:09.927145 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:09.927107    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjsdr\" (UniqueName: \"kubernetes.io/projected/23a43544-b4a0-4a97-a686-1bdf53299765-kube-api-access-vjsdr\") pod \"payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm\" (UID: \"23a43544-b4a0-4a97-a686-1bdf53299765\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"
Nov 02 13:39:10.004079 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:10.004047    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"
Nov 02 13:39:10.123071 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:10.123040    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"
Nov 02 13:39:10.378486 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:10.378458    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"]
Nov 02 13:39:10.379171 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:39:10.379145    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a43544_b4a0_4a97_a686_1bdf53299765.slice/crio-5b9ad3b1e9fd6b29e31e4f1f4f0799c7455e6bb6cda1872232ea19f6a40652cb WatchSource:0}: Error finding container 5b9ad3b1e9fd6b29e31e4f1f4f0799c7455e6bb6cda1872232ea19f6a40652cb: Status 404 returned error can't find the container with id 5b9ad3b1e9fd6b29e31e4f1f4f0799c7455e6bb6cda1872232ea19f6a40652cb
Nov 02 13:39:10.498813 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:10.498783    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"]
Nov 02 13:39:10.500108 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:39:10.500084    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d0ec52_b21e_484d_9eeb_04ccb55bd150.slice/crio-6ec440d4de4b28e485510aec03bbf1bf9fca798008e93b912ab45560d139914d WatchSource:0}: Error finding container 6ec440d4de4b28e485510aec03bbf1bf9fca798008e93b912ab45560d139914d: Status 404 returned error can't find the container with id 6ec440d4de4b28e485510aec03bbf1bf9fca798008e93b912ab45560d139914d
Nov 02 13:39:10.759244 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:10.759200    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm" event={"ID":"23a43544-b4a0-4a97-a686-1bdf53299765","Type":"ContainerStarted","Data":"adcb92e4ed8d0f944ce3dd31865925147205c9e594ba680cb6979aa4318b2d03"}
Nov 02 13:39:10.759716 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:10.759393    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"
Nov 02 13:39:10.759716 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:10.759428    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm" event={"ID":"23a43544-b4a0-4a97-a686-1bdf53299765","Type":"ContainerStarted","Data":"5b9ad3b1e9fd6b29e31e4f1f4f0799c7455e6bb6cda1872232ea19f6a40652cb"}
Nov 02 13:39:10.760838 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:10.760813    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4" event={"ID":"a0d0ec52-b21e-484d-9eeb-04ccb55bd150","Type":"ContainerStarted","Data":"922561dad1b5e5c30029230043c73a86a3578c40f017fa5130c23fecfa6083bc"}
Nov 02 13:39:10.760950 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:10.760852    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4" event={"ID":"a0d0ec52-b21e-484d-9eeb-04ccb55bd150","Type":"ContainerStarted","Data":"6ec440d4de4b28e485510aec03bbf1bf9fca798008e93b912ab45560d139914d"}
Nov 02 13:39:10.760950 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:10.760882    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"
Nov 02 13:39:10.877542 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:10.877490    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm" podStartSLOduration=1.8774760640000001 podStartE2EDuration="1.877476064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:39:10.877224945 +0000 UTC m=+6179.440933365" watchObservedRunningTime="2026-11-02 13:39:10.877476064 +0000 UTC m=+6179.441184516"
Nov 02 13:39:11.004161 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:11.004104    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4" podStartSLOduration=2.004084443 podStartE2EDuration="2.004084443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:39:10.999643153 +0000 UTC m=+6179.563351589" watchObservedRunningTime="2026-11-02 13:39:11.004084443 +0000 UTC m=+6179.567792853"
Nov 02 13:39:15.141538 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:15.141493    4464 scope.go:122] "RemoveContainer" containerID="f177c825675244cbe806b558fb63a652b53c09975b81db7cba5894f7864c4275"
Nov 02 13:39:15.150527 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:15.150507    4464 scope.go:122] "RemoveContainer" containerID="80f0c8bf5ace0e9b3a5693eaf79d08636d8cdfcee3bbfa5870bd545601623457"
Nov 02 13:39:15.158318 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:15.158301    4464 scope.go:122] "RemoveContainer" containerID="e69a39caa238347be560f57543947fa9ac79b53f7ae63e57a9d2a6e73a3c8f4e"
Nov 02 13:39:16.783761 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:16.783706    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:39:20.435072 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:20.435041    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"]
Nov 02 13:39:20.440369 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:20.440352    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"
Nov 02 13:39:20.497585 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:20.497556    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"]
Nov 02 13:39:20.544343 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:20.544315    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:39:20.544512 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:20.544377    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:39:20.544512 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:20.544383    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:39:20.544512 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:20.544420    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:39:20.582546 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:20.582520    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33dd7cb2-3a81-4cee-ab19-0e25720946e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt\" (UID: \"33dd7cb2-3a81-4cee-ab19-0e25720946e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"
Nov 02 13:39:20.582666 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:20.582651    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5npf\" (UniqueName: \"kubernetes.io/projected/33dd7cb2-3a81-4cee-ab19-0e25720946e4-kube-api-access-b5npf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt\" (UID: \"33dd7cb2-3a81-4cee-ab19-0e25720946e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"
Nov 02 13:39:20.582720 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:20.582688    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33dd7cb2-3a81-4cee-ab19-0e25720946e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt\" (UID: \"33dd7cb2-3a81-4cee-ab19-0e25720946e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"
Nov 02 13:39:20.683658 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:20.683627    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5npf\" (UniqueName: \"kubernetes.io/projected/33dd7cb2-3a81-4cee-ab19-0e25720946e4-kube-api-access-b5npf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt\" (UID: \"33dd7cb2-3a81-4cee-ab19-0e25720946e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"
Nov 02 13:39:20.683815 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:20.683674    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33dd7cb2-3a81-4cee-ab19-0e25720946e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt\" (UID: \"33dd7cb2-3a81-4cee-ab19-0e25720946e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"
Nov 02 13:39:20.683889 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:20.683864    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33dd7cb2-3a81-4cee-ab19-0e25720946e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt\" (UID: \"33dd7cb2-3a81-4cee-ab19-0e25720946e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"
Nov 02 13:39:20.684376 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:20.684357    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33dd7cb2-3a81-4cee-ab19-0e25720946e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt\" (UID: \"33dd7cb2-3a81-4cee-ab19-0e25720946e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"
Nov 02 13:39:20.686279 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:20.686238    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33dd7cb2-3a81-4cee-ab19-0e25720946e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt\" (UID: \"33dd7cb2-3a81-4cee-ab19-0e25720946e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"
Nov 02 13:39:20.745910 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:20.745888    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5npf\" (UniqueName: \"kubernetes.io/projected/33dd7cb2-3a81-4cee-ab19-0e25720946e4-kube-api-access-b5npf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt\" (UID: \"33dd7cb2-3a81-4cee-ab19-0e25720946e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"
Nov 02 13:39:20.752731 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:20.752709    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"
Nov 02 13:39:21.124677 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:21.124651    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"]
Nov 02 13:39:21.126855 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:39:21.126829    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33dd7cb2_3a81_4cee_ab19_0e25720946e4.slice/crio-cc1770b525d815a732ae28a4f4af9d646cf756f3d6a21484379f1a8a5306f45a WatchSource:0}: Error finding container cc1770b525d815a732ae28a4f4af9d646cf756f3d6a21484379f1a8a5306f45a: Status 404 returned error can't find the container with id cc1770b525d815a732ae28a4f4af9d646cf756f3d6a21484379f1a8a5306f45a
Nov 02 13:39:21.769887 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:21.769851    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"
Nov 02 13:39:21.770321 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:21.769994    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"
Nov 02 13:39:21.826651 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:21.826607    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt" event={"ID":"33dd7cb2-3a81-4cee-ab19-0e25720946e4","Type":"ContainerStarted","Data":"378f58e95870a9a0484ef33313fa06ac8bbc560a53cf0b797f1f4cdb60e78358"}
Nov 02 13:39:21.826841 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:21.826667    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt" event={"ID":"33dd7cb2-3a81-4cee-ab19-0e25720946e4","Type":"ContainerStarted","Data":"cc1770b525d815a732ae28a4f4af9d646cf756f3d6a21484379f1a8a5306f45a"}
Nov 02 13:39:22.125527 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:22.125469    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt" podStartSLOduration=2.125455042 podStartE2EDuration="2.125455042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:39:22.122966298 +0000 UTC m=+6190.686674708" watchObservedRunningTime="2026-11-02 13:39:22.125455042 +0000 UTC m=+6190.689163479"
Nov 02 13:39:22.834494 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:22.834454    4464 generic.go:356] "Generic (PLEG): container finished" podID="33dd7cb2-3a81-4cee-ab19-0e25720946e4" containerID="378f58e95870a9a0484ef33313fa06ac8bbc560a53cf0b797f1f4cdb60e78358" exitCode=0
Nov 02 13:39:22.834882 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:22.834535    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt" event={"ID":"33dd7cb2-3a81-4cee-ab19-0e25720946e4","Type":"ContainerDied","Data":"378f58e95870a9a0484ef33313fa06ac8bbc560a53cf0b797f1f4cdb60e78358"}
Nov 02 13:39:23.977612 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:23.977587    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"
Nov 02 13:39:24.122397 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:24.122321    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33dd7cb2-3a81-4cee-ab19-0e25720946e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33dd7cb2-3a81-4cee-ab19-0e25720946e4-maas-api-xks-ca\") pod \"33dd7cb2-3a81-4cee-ab19-0e25720946e4\" (UID: \"33dd7cb2-3a81-4cee-ab19-0e25720946e4\") "
Nov 02 13:39:24.122543 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:24.122402    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33dd7cb2-3a81-4cee-ab19-0e25720946e4-kube-api-access-b5npf\" (UniqueName: \"kubernetes.io/projected/33dd7cb2-3a81-4cee-ab19-0e25720946e4-kube-api-access-b5npf\") pod \"33dd7cb2-3a81-4cee-ab19-0e25720946e4\" (UID: \"33dd7cb2-3a81-4cee-ab19-0e25720946e4\") "
Nov 02 13:39:24.122543 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:24.122483    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33dd7cb2-3a81-4cee-ab19-0e25720946e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33dd7cb2-3a81-4cee-ab19-0e25720946e4-maas-api-openshift-ca\") pod \"33dd7cb2-3a81-4cee-ab19-0e25720946e4\" (UID: \"33dd7cb2-3a81-4cee-ab19-0e25720946e4\") "
Nov 02 13:39:24.122855 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:24.122831    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33dd7cb2-3a81-4cee-ab19-0e25720946e4-maas-api-openshift-ca" pod "33dd7cb2-3a81-4cee-ab19-0e25720946e4" (UID: "33dd7cb2-3a81-4cee-ab19-0e25720946e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:39:24.124566 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:24.124539    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33dd7cb2-3a81-4cee-ab19-0e25720946e4-maas-api-xks-ca" pod "33dd7cb2-3a81-4cee-ab19-0e25720946e4" (UID: "33dd7cb2-3a81-4cee-ab19-0e25720946e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:39:24.124643 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:24.124564    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33dd7cb2-3a81-4cee-ab19-0e25720946e4-kube-api-access-b5npf" pod "33dd7cb2-3a81-4cee-ab19-0e25720946e4" (UID: "33dd7cb2-3a81-4cee-ab19-0e25720946e4"). InnerVolumeSpecName "kube-api-access-b5npf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:39:24.223524 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:24.223490    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33dd7cb2-3a81-4cee-ab19-0e25720946e4-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:24.223524 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:24.223521    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5npf\" (UniqueName: \"kubernetes.io/projected/33dd7cb2-3a81-4cee-ab19-0e25720946e4-kube-api-access-b5npf\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:24.223524 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:24.223531    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33dd7cb2-3a81-4cee-ab19-0e25720946e4-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:24.846194 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:24.846168    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"
Nov 02 13:39:24.846370 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:24.846198    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt" event={"ID":"33dd7cb2-3a81-4cee-ab19-0e25720946e4","Type":"ContainerDied","Data":"cc1770b525d815a732ae28a4f4af9d646cf756f3d6a21484379f1a8a5306f45a"}
Nov 02 13:39:24.846370 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:24.846248    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc1770b525d815a732ae28a4f4af9d646cf756f3d6a21484379f1a8a5306f45a"
Nov 02 13:39:28.291621 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:39:28.291586    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:39:30.047804 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:30.047742    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"]
Nov 02 13:39:30.098260 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:30.098227    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-bbrcr"]
Nov 02 13:39:31.581224 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:31.581197    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3418026-660d-4af2-a0ea-5aec83c8f0bc" path="/var/lib/kubelet/pods/a3418026-660d-4af2-a0ea-5aec83c8f0bc/volumes"
Nov 02 13:39:34.204439 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:34.204403    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"]
Nov 02 13:39:34.318861 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:34.318830    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"]
Nov 02 13:39:39.205299 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.205240    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm" podUID="23a43544-b4a0-4a97-a686-1bdf53299765" containerName="payload-processing" containerID="cri-o://adcb92e4ed8d0f944ce3dd31865925147205c9e594ba680cb6979aa4318b2d03" gracePeriod=25
Nov 02 13:39:39.319789 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.319731    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4" podUID="a0d0ec52-b21e-484d-9eeb-04ccb55bd150" containerName="payload-pre-processing" containerID="cri-o://922561dad1b5e5c30029230043c73a86a3578c40f017fa5130c23fecfa6083bc" gracePeriod=25
Nov 02 13:39:39.472587 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.472560    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"
Nov 02 13:39:39.582928 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.582898    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"
Nov 02 13:39:39.594785 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.594759    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a43544-b4a0-4a97-a686-1bdf53299765-kube-api-access-vjsdr\" (UniqueName: \"kubernetes.io/projected/23a43544-b4a0-4a97-a686-1bdf53299765-kube-api-access-vjsdr\") pod \"23a43544-b4a0-4a97-a686-1bdf53299765\" (UID: \"23a43544-b4a0-4a97-a686-1bdf53299765\") "
Nov 02 13:39:39.594966 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.594949    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a43544-b4a0-4a97-a686-1bdf53299765-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a43544-b4a0-4a97-a686-1bdf53299765-plugins-config-volume\") pod \"23a43544-b4a0-4a97-a686-1bdf53299765\" (UID: \"23a43544-b4a0-4a97-a686-1bdf53299765\") "
Nov 02 13:39:39.595381 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.595356    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a43544-b4a0-4a97-a686-1bdf53299765-plugins-config-volume" pod "23a43544-b4a0-4a97-a686-1bdf53299765" (UID: "23a43544-b4a0-4a97-a686-1bdf53299765"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:39:39.595554 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.595536    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a43544-b4a0-4a97-a686-1bdf53299765-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:39.597365 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.597338    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a43544-b4a0-4a97-a686-1bdf53299765-kube-api-access-vjsdr" pod "23a43544-b4a0-4a97-a686-1bdf53299765" (UID: "23a43544-b4a0-4a97-a686-1bdf53299765"). InnerVolumeSpecName "kube-api-access-vjsdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:39:39.686203 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.686168    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"]
Nov 02 13:39:39.689335 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.689313    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7ddf45-57fgt"]
Nov 02 13:39:39.696539 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.696515    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d0ec52-b21e-484d-9eeb-04ccb55bd150-kube-api-access-8j8jf\" (UniqueName: \"kubernetes.io/projected/a0d0ec52-b21e-484d-9eeb-04ccb55bd150-kube-api-access-8j8jf\") pod \"a0d0ec52-b21e-484d-9eeb-04ccb55bd150\" (UID: \"a0d0ec52-b21e-484d-9eeb-04ccb55bd150\") "
Nov 02 13:39:39.696636 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.696545    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d0ec52-b21e-484d-9eeb-04ccb55bd150-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0d0ec52-b21e-484d-9eeb-04ccb55bd150-plugins-config-volume\") pod \"a0d0ec52-b21e-484d-9eeb-04ccb55bd150\" (UID: \"a0d0ec52-b21e-484d-9eeb-04ccb55bd150\") "
Nov 02 13:39:39.696840 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.696821    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjsdr\" (UniqueName: \"kubernetes.io/projected/23a43544-b4a0-4a97-a686-1bdf53299765-kube-api-access-vjsdr\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:39.697002 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.696957    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d0ec52-b21e-484d-9eeb-04ccb55bd150-plugins-config-volume" pod "a0d0ec52-b21e-484d-9eeb-04ccb55bd150" (UID: "a0d0ec52-b21e-484d-9eeb-04ccb55bd150"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:39:39.698532 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.698511    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d0ec52-b21e-484d-9eeb-04ccb55bd150-kube-api-access-8j8jf" pod "a0d0ec52-b21e-484d-9eeb-04ccb55bd150" (UID: "a0d0ec52-b21e-484d-9eeb-04ccb55bd150"). InnerVolumeSpecName "kube-api-access-8j8jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:39:39.797448 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.797419    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j8jf\" (UniqueName: \"kubernetes.io/projected/a0d0ec52-b21e-484d-9eeb-04ccb55bd150-kube-api-access-8j8jf\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:39.797448 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.797446    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0d0ec52-b21e-484d-9eeb-04ccb55bd150-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:39.909772 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.909740    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"]
Nov 02 13:39:39.910033 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.910008    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml" podUID="dff4f721-0c05-44d9-acff-9d121302d980" containerName="istio-proxy" containerID="cri-o://8d078196ecccb493c60c190808da82bc053062feaf79b5730d60517b81a98bc0" gracePeriod=30
Nov 02 13:39:39.944903 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.944871    4464 generic.go:356] "Generic (PLEG): container finished" podID="23a43544-b4a0-4a97-a686-1bdf53299765" containerID="adcb92e4ed8d0f944ce3dd31865925147205c9e594ba680cb6979aa4318b2d03" exitCode=0
Nov 02 13:39:39.945105 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.944922    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"
Nov 02 13:39:39.945105 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.944953    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm" event={"ID":"23a43544-b4a0-4a97-a686-1bdf53299765","Type":"ContainerDied","Data":"adcb92e4ed8d0f944ce3dd31865925147205c9e594ba680cb6979aa4318b2d03"}
Nov 02 13:39:39.945105 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.945028    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm" event={"ID":"23a43544-b4a0-4a97-a686-1bdf53299765","Type":"ContainerDied","Data":"5b9ad3b1e9fd6b29e31e4f1f4f0799c7455e6bb6cda1872232ea19f6a40652cb"}
Nov 02 13:39:39.945105 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.945052    4464 scope.go:122] "RemoveContainer" containerID="adcb92e4ed8d0f944ce3dd31865925147205c9e594ba680cb6979aa4318b2d03"
Nov 02 13:39:39.946612 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.946591    4464 generic.go:356] "Generic (PLEG): container finished" podID="a0d0ec52-b21e-484d-9eeb-04ccb55bd150" containerID="922561dad1b5e5c30029230043c73a86a3578c40f017fa5130c23fecfa6083bc" exitCode=0
Nov 02 13:39:39.946739 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.946631    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4" event={"ID":"a0d0ec52-b21e-484d-9eeb-04ccb55bd150","Type":"ContainerDied","Data":"922561dad1b5e5c30029230043c73a86a3578c40f017fa5130c23fecfa6083bc"}
Nov 02 13:39:39.946739 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.946659    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"
Nov 02 13:39:39.946739 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.946682    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4" event={"ID":"a0d0ec52-b21e-484d-9eeb-04ccb55bd150","Type":"ContainerDied","Data":"6ec440d4de4b28e485510aec03bbf1bf9fca798008e93b912ab45560d139914d"}
Nov 02 13:39:39.954671 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.954651    4464 scope.go:122] "RemoveContainer" containerID="adcb92e4ed8d0f944ce3dd31865925147205c9e594ba680cb6979aa4318b2d03"
Nov 02 13:39:39.954944 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:39:39.954919    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adcb92e4ed8d0f944ce3dd31865925147205c9e594ba680cb6979aa4318b2d03\": container with ID starting with adcb92e4ed8d0f944ce3dd31865925147205c9e594ba680cb6979aa4318b2d03 not found: ID does not exist" containerID="adcb92e4ed8d0f944ce3dd31865925147205c9e594ba680cb6979aa4318b2d03"
Nov 02 13:39:39.955053 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.954956    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adcb92e4ed8d0f944ce3dd31865925147205c9e594ba680cb6979aa4318b2d03"} err="failed to get container status \"adcb92e4ed8d0f944ce3dd31865925147205c9e594ba680cb6979aa4318b2d03\": rpc error: code = NotFound desc = could not find container \"adcb92e4ed8d0f944ce3dd31865925147205c9e594ba680cb6979aa4318b2d03\": container with ID starting with adcb92e4ed8d0f944ce3dd31865925147205c9e594ba680cb6979aa4318b2d03 not found: ID does not exist"
Nov 02 13:39:39.955053 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.955011    4464 scope.go:122] "RemoveContainer" containerID="922561dad1b5e5c30029230043c73a86a3578c40f017fa5130c23fecfa6083bc"
Nov 02 13:39:39.964373 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.964353    4464 scope.go:122] "RemoveContainer" containerID="922561dad1b5e5c30029230043c73a86a3578c40f017fa5130c23fecfa6083bc"
Nov 02 13:39:39.964697 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:39:39.964662    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922561dad1b5e5c30029230043c73a86a3578c40f017fa5130c23fecfa6083bc\": container with ID starting with 922561dad1b5e5c30029230043c73a86a3578c40f017fa5130c23fecfa6083bc not found: ID does not exist" containerID="922561dad1b5e5c30029230043c73a86a3578c40f017fa5130c23fecfa6083bc"
Nov 02 13:39:39.964774 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:39.964694    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922561dad1b5e5c30029230043c73a86a3578c40f017fa5130c23fecfa6083bc"} err="failed to get container status \"922561dad1b5e5c30029230043c73a86a3578c40f017fa5130c23fecfa6083bc\": rpc error: code = NotFound desc = could not find container \"922561dad1b5e5c30029230043c73a86a3578c40f017fa5130c23fecfa6083bc\": container with ID starting with 922561dad1b5e5c30029230043c73a86a3578c40f017fa5130c23fecfa6083bc not found: ID does not exist"
Nov 02 13:39:40.152568 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:40.152485    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"]
Nov 02 13:39:40.152802 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:40.152786    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7ddf45-567b8bfd668dzm"]
Nov 02 13:39:40.259590 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:40.259559    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"]
Nov 02 13:39:40.265596 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:40.265566    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7ddf45-5d887486d4"]
Nov 02 13:39:41.580716 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.580683    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a43544-b4a0-4a97-a686-1bdf53299765" path="/var/lib/kubelet/pods/23a43544-b4a0-4a97-a686-1bdf53299765/volumes"
Nov 02 13:39:41.581176 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.581162    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33dd7cb2-3a81-4cee-ab19-0e25720946e4" path="/var/lib/kubelet/pods/33dd7cb2-3a81-4cee-ab19-0e25720946e4/volumes"
Nov 02 13:39:41.581544 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.581528    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d0ec52-b21e-484d-9eeb-04ccb55bd150" path="/var/lib/kubelet/pods/a0d0ec52-b21e-484d-9eeb-04ccb55bd150/volumes"
Nov 02 13:39:41.596992 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.596955    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"]
Nov 02 13:39:41.647929 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.647904    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-vf4vt"]
Nov 02 13:39:41.726309 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.726280    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"]
Nov 02 13:39:41.726704 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.726691    4464 cpu_manager.go:414] "containerMap: removing container" podUID="33dd7cb2-3a81-4cee-ab19-0e25720946e4" containerName="revoke-keys"
Nov 02 13:39:41.726748 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.726705    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33dd7cb2-3a81-4cee-ab19-0e25720946e4" containerName="revoke-keys"
Nov 02 13:39:41.726748 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.726738    4464 cpu_manager.go:414] "containerMap: removing container" podUID="23a43544-b4a0-4a97-a686-1bdf53299765" containerName="payload-processing"
Nov 02 13:39:41.726748 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.726743    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a43544-b4a0-4a97-a686-1bdf53299765" containerName="payload-processing"
Nov 02 13:39:41.726854 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.726751    4464 cpu_manager.go:414] "containerMap: removing container" podUID="a0d0ec52-b21e-484d-9eeb-04ccb55bd150" containerName="payload-pre-processing"
Nov 02 13:39:41.726854 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.726756    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d0ec52-b21e-484d-9eeb-04ccb55bd150" containerName="payload-pre-processing"
Nov 02 13:39:41.726854 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.726818    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d0ec52-b21e-484d-9eeb-04ccb55bd150" containerName="payload-pre-processing"
Nov 02 13:39:41.726854 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.726828    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="33dd7cb2-3a81-4cee-ab19-0e25720946e4" containerName="revoke-keys"
Nov 02 13:39:41.726854 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.726841    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a43544-b4a0-4a97-a686-1bdf53299765" containerName="payload-processing"
Nov 02 13:39:41.731334 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.731318    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"
Nov 02 13:39:41.786622 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.786579    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:39:41.786892 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.786868    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:39:41.787385 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.786907    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:39:41.787385 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.786932    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:39:41.790265 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.790245    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"]
Nov 02 13:39:41.815685 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.815662    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbl72\" (UniqueName: \"kubernetes.io/projected/e0eb3d1d-b80e-428e-bd76-9e7d70438152-kube-api-access-pbl72\") pod \"maas-api-revoke-sub-3d4156f1-wgfxn\" (UID: \"e0eb3d1d-b80e-428e-bd76-9e7d70438152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"
Nov 02 13:39:41.815811 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.815704    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0eb3d1d-b80e-428e-bd76-9e7d70438152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wgfxn\" (UID: \"e0eb3d1d-b80e-428e-bd76-9e7d70438152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"
Nov 02 13:39:41.815811 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.815727    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0eb3d1d-b80e-428e-bd76-9e7d70438152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wgfxn\" (UID: \"e0eb3d1d-b80e-428e-bd76-9e7d70438152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"
Nov 02 13:39:41.917248 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.917179    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbl72\" (UniqueName: \"kubernetes.io/projected/e0eb3d1d-b80e-428e-bd76-9e7d70438152-kube-api-access-pbl72\") pod \"maas-api-revoke-sub-3d4156f1-wgfxn\" (UID: \"e0eb3d1d-b80e-428e-bd76-9e7d70438152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"
Nov 02 13:39:41.917248 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.917227    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0eb3d1d-b80e-428e-bd76-9e7d70438152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wgfxn\" (UID: \"e0eb3d1d-b80e-428e-bd76-9e7d70438152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"
Nov 02 13:39:41.917421 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.917263    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0eb3d1d-b80e-428e-bd76-9e7d70438152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wgfxn\" (UID: \"e0eb3d1d-b80e-428e-bd76-9e7d70438152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"
Nov 02 13:39:41.917941 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.917919    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0eb3d1d-b80e-428e-bd76-9e7d70438152-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wgfxn\" (UID: \"e0eb3d1d-b80e-428e-bd76-9e7d70438152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"
Nov 02 13:39:41.919570 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.919546    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0eb3d1d-b80e-428e-bd76-9e7d70438152-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wgfxn\" (UID: \"e0eb3d1d-b80e-428e-bd76-9e7d70438152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"
Nov 02 13:39:41.978851 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:41.978825    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbl72\" (UniqueName: \"kubernetes.io/projected/e0eb3d1d-b80e-428e-bd76-9e7d70438152-kube-api-access-pbl72\") pod \"maas-api-revoke-sub-3d4156f1-wgfxn\" (UID: \"e0eb3d1d-b80e-428e-bd76-9e7d70438152\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"
Nov 02 13:39:42.043523 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:42.043500    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"
Nov 02 13:39:42.169203 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:42.169134    4464 patch_prober.go:29] interesting pod/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:39:42.169337 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:42.169205    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml" podUID="dff4f721-0c05-44d9-acff-9d121302d980" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:39:42.417718 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:42.417690    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"]
Nov 02 13:39:42.418042 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:39:42.418016    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0eb3d1d_b80e_428e_bd76_9e7d70438152.slice/crio-bcc4cc53ede15fb601e324aa44654fcb80fa2d7f47f27c564fc5a57d3ae9f348 WatchSource:0}: Error finding container bcc4cc53ede15fb601e324aa44654fcb80fa2d7f47f27c564fc5a57d3ae9f348: Status 404 returned error can't find the container with id bcc4cc53ede15fb601e324aa44654fcb80fa2d7f47f27c564fc5a57d3ae9f348
Nov 02 13:39:42.965581 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:42.965544    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn" event={"ID":"e0eb3d1d-b80e-428e-bd76-9e7d70438152","Type":"ContainerStarted","Data":"60de7e10367d430ffc04cefbf495227437002d740e6e56b8aedf65dd28124a65"}
Nov 02 13:39:42.965993 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:42.965606    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn" event={"ID":"e0eb3d1d-b80e-428e-bd76-9e7d70438152","Type":"ContainerStarted","Data":"bcc4cc53ede15fb601e324aa44654fcb80fa2d7f47f27c564fc5a57d3ae9f348"}
Nov 02 13:39:43.084719 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:43.084669    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn" podStartSLOduration=2.084655299 podStartE2EDuration="2.084655299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:39:43.082640546 +0000 UTC m=+6211.646348959" watchObservedRunningTime="2026-11-02 13:39:43.084655299 +0000 UTC m=+6211.648363709"
Nov 02 13:39:43.580824 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:43.580754    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd739ad-127a-48f4-b1fa-73fc62634496" path="/var/lib/kubelet/pods/5dd739ad-127a-48f4-b1fa-73fc62634496/volumes"
Nov 02 13:39:43.708846 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:43.708811    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:39:43.972359 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:43.972326    4464 generic.go:356] "Generic (PLEG): container finished" podID="e0eb3d1d-b80e-428e-bd76-9e7d70438152" containerID="60de7e10367d430ffc04cefbf495227437002d740e6e56b8aedf65dd28124a65" exitCode=0
Nov 02 13:39:43.972751 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:43.972415    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn" event={"ID":"e0eb3d1d-b80e-428e-bd76-9e7d70438152","Type":"ContainerDied","Data":"60de7e10367d430ffc04cefbf495227437002d740e6e56b8aedf65dd28124a65"}
Nov 02 13:39:45.121666 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.121636    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"
Nov 02 13:39:45.176433 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.176379    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:39:45.250770 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.250739    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dff4f721-0c05-44d9-acff-9d121302d980-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff4f721-0c05-44d9-acff-9d121302d980-istio-podinfo\") pod \"dff4f721-0c05-44d9-acff-9d121302d980\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") "
Nov 02 13:39:45.250904 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.250806    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0eb3d1d-b80e-428e-bd76-9e7d70438152-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0eb3d1d-b80e-428e-bd76-9e7d70438152-maas-api-openshift-ca\") pod \"e0eb3d1d-b80e-428e-bd76-9e7d70438152\" (UID: \"e0eb3d1d-b80e-428e-bd76-9e7d70438152\") "
Nov 02 13:39:45.250904 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.250847    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff4f721-0c05-44d9-acff-9d121302d980-istio-token\" (UniqueName: \"kubernetes.io/projected/dff4f721-0c05-44d9-acff-9d121302d980-istio-token\") pod \"dff4f721-0c05-44d9-acff-9d121302d980\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") "
Nov 02 13:39:45.250904 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.250892    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff4f721-0c05-44d9-acff-9d121302d980-kube-api-access-6bzdm\" (UniqueName: \"kubernetes.io/projected/dff4f721-0c05-44d9-acff-9d121302d980-kube-api-access-6bzdm\") pod \"dff4f721-0c05-44d9-acff-9d121302d980\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") "
Nov 02 13:39:45.251110 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.250924    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0eb3d1d-b80e-428e-bd76-9e7d70438152-kube-api-access-pbl72\" (UniqueName: \"kubernetes.io/projected/e0eb3d1d-b80e-428e-bd76-9e7d70438152-kube-api-access-pbl72\") pod \"e0eb3d1d-b80e-428e-bd76-9e7d70438152\" (UID: \"e0eb3d1d-b80e-428e-bd76-9e7d70438152\") "
Nov 02 13:39:45.251110 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.250961    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-istio-data\") pod \"dff4f721-0c05-44d9-acff-9d121302d980\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") "
Nov 02 13:39:45.251110 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.251008    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-workload-socket\") pod \"dff4f721-0c05-44d9-acff-9d121302d980\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") "
Nov 02 13:39:45.251110 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.251091    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-credential-socket\") pod \"dff4f721-0c05-44d9-acff-9d121302d980\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") "
Nov 02 13:39:45.251343 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.251120    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-workload-certs\") pod \"dff4f721-0c05-44d9-acff-9d121302d980\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") "
Nov 02 13:39:45.251343 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.251154    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0eb3d1d-b80e-428e-bd76-9e7d70438152-maas-api-openshift-ca" pod "e0eb3d1d-b80e-428e-bd76-9e7d70438152" (UID: "e0eb3d1d-b80e-428e-bd76-9e7d70438152"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:39:45.251343 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.251172    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0eb3d1d-b80e-428e-bd76-9e7d70438152-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0eb3d1d-b80e-428e-bd76-9e7d70438152-maas-api-xks-ca\") pod \"e0eb3d1d-b80e-428e-bd76-9e7d70438152\" (UID: \"e0eb3d1d-b80e-428e-bd76-9e7d70438152\") "
Nov 02 13:39:45.251343 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.251203    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dff4f721-0c05-44d9-acff-9d121302d980-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff4f721-0c05-44d9-acff-9d121302d980-istiod-ca-cert\") pod \"dff4f721-0c05-44d9-acff-9d121302d980\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") "
Nov 02 13:39:45.251343 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.251230    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-istio-envoy\") pod \"dff4f721-0c05-44d9-acff-9d121302d980\" (UID: \"dff4f721-0c05-44d9-acff-9d121302d980\") "
Nov 02 13:39:45.251612 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.251386    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-istio-data" pod "dff4f721-0c05-44d9-acff-9d121302d980" (UID: "dff4f721-0c05-44d9-acff-9d121302d980"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:39:45.251612 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.251435    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-workload-socket" pod "dff4f721-0c05-44d9-acff-9d121302d980" (UID: "dff4f721-0c05-44d9-acff-9d121302d980"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:39:45.251612 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.251499    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-workload-certs" pod "dff4f721-0c05-44d9-acff-9d121302d980" (UID: "dff4f721-0c05-44d9-acff-9d121302d980"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:39:45.251773 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.251647    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-credential-socket" pod "dff4f721-0c05-44d9-acff-9d121302d980" (UID: "dff4f721-0c05-44d9-acff-9d121302d980"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:39:45.251773 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.251722    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:45.251773 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.251745    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:45.251773 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.251760    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0eb3d1d-b80e-428e-bd76-9e7d70438152-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:45.252024 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.251775    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:45.252024 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.251790    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:45.252024 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.251938    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff4f721-0c05-44d9-acff-9d121302d980-istiod-ca-cert" pod "dff4f721-0c05-44d9-acff-9d121302d980" (UID: "dff4f721-0c05-44d9-acff-9d121302d980"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:39:45.253657 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.253627    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff4f721-0c05-44d9-acff-9d121302d980-istio-token" pod "dff4f721-0c05-44d9-acff-9d121302d980" (UID: "dff4f721-0c05-44d9-acff-9d121302d980"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:39:45.253767 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.253718    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff4f721-0c05-44d9-acff-9d121302d980-kube-api-access-6bzdm" pod "dff4f721-0c05-44d9-acff-9d121302d980" (UID: "dff4f721-0c05-44d9-acff-9d121302d980"). InnerVolumeSpecName "kube-api-access-6bzdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:39:45.253826 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.253792    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0eb3d1d-b80e-428e-bd76-9e7d70438152-kube-api-access-pbl72" pod "e0eb3d1d-b80e-428e-bd76-9e7d70438152" (UID: "e0eb3d1d-b80e-428e-bd76-9e7d70438152"). InnerVolumeSpecName "kube-api-access-pbl72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:39:45.254139 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.254121    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dff4f721-0c05-44d9-acff-9d121302d980-istio-podinfo" pod "dff4f721-0c05-44d9-acff-9d121302d980" (UID: "dff4f721-0c05-44d9-acff-9d121302d980"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:39:45.254139 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.254127    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-istio-envoy" pod "dff4f721-0c05-44d9-acff-9d121302d980" (UID: "dff4f721-0c05-44d9-acff-9d121302d980"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:39:45.254258 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.254199    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0eb3d1d-b80e-428e-bd76-9e7d70438152-maas-api-xks-ca" pod "e0eb3d1d-b80e-428e-bd76-9e7d70438152" (UID: "e0eb3d1d-b80e-428e-bd76-9e7d70438152"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:39:45.353159 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.353136    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff4f721-0c05-44d9-acff-9d121302d980-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:45.353159 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.353157    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bzdm\" (UniqueName: \"kubernetes.io/projected/dff4f721-0c05-44d9-acff-9d121302d980-kube-api-access-6bzdm\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:45.353327 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.353167    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbl72\" (UniqueName: \"kubernetes.io/projected/e0eb3d1d-b80e-428e-bd76-9e7d70438152-kube-api-access-pbl72\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:45.353327 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.353177    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0eb3d1d-b80e-428e-bd76-9e7d70438152-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:45.353327 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.353185    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff4f721-0c05-44d9-acff-9d121302d980-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:45.353327 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.353194    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff4f721-0c05-44d9-acff-9d121302d980-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:45.353327 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.353202    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff4f721-0c05-44d9-acff-9d121302d980-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:39:45.985828 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.985794    4464 generic.go:356] "Generic (PLEG): container finished" podID="dff4f721-0c05-44d9-acff-9d121302d980" containerID="8d078196ecccb493c60c190808da82bc053062feaf79b5730d60517b81a98bc0" exitCode=0
Nov 02 13:39:45.986024 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.985856    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"
Nov 02 13:39:45.986024 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.985873    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml" event={"ID":"dff4f721-0c05-44d9-acff-9d121302d980","Type":"ContainerDied","Data":"8d078196ecccb493c60c190808da82bc053062feaf79b5730d60517b81a98bc0"}
Nov 02 13:39:45.986024 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.985935    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml" event={"ID":"dff4f721-0c05-44d9-acff-9d121302d980","Type":"ContainerDied","Data":"2539e62fd526e64354520df4142972e304dbcd305b152172baac60ab40dc9c06"}
Nov 02 13:39:45.986024 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.985971    4464 scope.go:122] "RemoveContainer" containerID="8d078196ecccb493c60c190808da82bc053062feaf79b5730d60517b81a98bc0"
Nov 02 13:39:45.987532 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.987454    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn" event={"ID":"e0eb3d1d-b80e-428e-bd76-9e7d70438152","Type":"ContainerDied","Data":"bcc4cc53ede15fb601e324aa44654fcb80fa2d7f47f27c564fc5a57d3ae9f348"}
Nov 02 13:39:45.987532 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.987501    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcc4cc53ede15fb601e324aa44654fcb80fa2d7f47f27c564fc5a57d3ae9f348"
Nov 02 13:39:45.987532 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.987467    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"
Nov 02 13:39:45.995532 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:45.995511    4464 scope.go:122] "RemoveContainer" containerID="10e8b7d6b2baf96844be9ace8454ab88e08acef01d664ba0a2d19c3a83009187"
Nov 02 13:39:46.003691 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:46.003675    4464 scope.go:122] "RemoveContainer" containerID="8d078196ecccb493c60c190808da82bc053062feaf79b5730d60517b81a98bc0"
Nov 02 13:39:46.003936 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:39:46.003921    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d078196ecccb493c60c190808da82bc053062feaf79b5730d60517b81a98bc0\": container with ID starting with 8d078196ecccb493c60c190808da82bc053062feaf79b5730d60517b81a98bc0 not found: ID does not exist" containerID="8d078196ecccb493c60c190808da82bc053062feaf79b5730d60517b81a98bc0"
Nov 02 13:39:46.004004 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:46.003946    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d078196ecccb493c60c190808da82bc053062feaf79b5730d60517b81a98bc0"} err="failed to get container status \"8d078196ecccb493c60c190808da82bc053062feaf79b5730d60517b81a98bc0\": rpc error: code = NotFound desc = could not find container \"8d078196ecccb493c60c190808da82bc053062feaf79b5730d60517b81a98bc0\": container with ID starting with 8d078196ecccb493c60c190808da82bc053062feaf79b5730d60517b81a98bc0 not found: ID does not exist"
Nov 02 13:39:46.004004 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:46.003960    4464 scope.go:122] "RemoveContainer" containerID="10e8b7d6b2baf96844be9ace8454ab88e08acef01d664ba0a2d19c3a83009187"
Nov 02 13:39:46.004188 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:39:46.004176    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e8b7d6b2baf96844be9ace8454ab88e08acef01d664ba0a2d19c3a83009187\": container with ID starting with 10e8b7d6b2baf96844be9ace8454ab88e08acef01d664ba0a2d19c3a83009187 not found: ID does not exist" containerID="10e8b7d6b2baf96844be9ace8454ab88e08acef01d664ba0a2d19c3a83009187"
Nov 02 13:39:46.004231 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:46.004192    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e8b7d6b2baf96844be9ace8454ab88e08acef01d664ba0a2d19c3a83009187"} err="failed to get container status \"10e8b7d6b2baf96844be9ace8454ab88e08acef01d664ba0a2d19c3a83009187\": rpc error: code = NotFound desc = could not find container \"10e8b7d6b2baf96844be9ace8454ab88e08acef01d664ba0a2d19c3a83009187\": container with ID starting with 10e8b7d6b2baf96844be9ace8454ab88e08acef01d664ba0a2d19c3a83009187 not found: ID does not exist"
Nov 02 13:39:46.271840 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:46.271807    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"]
Nov 02 13:39:46.322310 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:46.322278    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-7ddf45-openshift-default-7db587fc85mjkml"]
Nov 02 13:39:47.048423 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:47.048391    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"]
Nov 02 13:39:47.050957 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:47.050936    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-fq8fl"]
Nov 02 13:39:47.581269 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:47.581239    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34cbe4ca-93a9-4410-9d83-9d82d7abc152" path="/var/lib/kubelet/pods/34cbe4ca-93a9-4410-9d83-9d82d7abc152/volumes"
Nov 02 13:39:47.581675 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:47.581662    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff4f721-0c05-44d9-acff-9d121302d980" path="/var/lib/kubelet/pods/dff4f721-0c05-44d9-acff-9d121302d980/volumes"
Nov 02 13:39:56.498541 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:56.498507    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:39:58.351315 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:39:58.351282    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:39:58.377999 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:58.377063    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-zfkls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:39:58.377999 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:58.377123    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls" podUID="85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:39:58.439121 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:58.439083    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-bp4h7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:39:58.439312 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:39:58.439147    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7" podUID="02c134c2-e2f8-4a16-a557-65786f86217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:40:15.193102 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:40:15.193070    4464 scope.go:122] "RemoveContainer" containerID="d294fc1517c7056f10ad547ee562a7b701edba959cf0b243821d981dc619a87b"
Nov 02 13:40:15.201390 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:40:15.201364    4464 scope.go:122] "RemoveContainer" containerID="f46cd496b6ad609f6e67cdee143ba6e277b4a066920bc4b3d09525fc7eabf51b"
Nov 02 13:40:28.291871 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:40:28.291785    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:40:58.291121 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:40:58.291084    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:41:11.048521 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:11.048476    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"]
Nov 02 13:41:11.099217 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:11.099185    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w7gtd"]
Nov 02 13:41:11.583743 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:11.583692    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90c23826-9769-4db4-b16b-9a4f329bc7cc" path="/var/lib/kubelet/pods/90c23826-9769-4db4-b16b-9a4f329bc7cc/volumes"
Nov 02 13:41:12.381926 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:12.381900    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:41:12.398111 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:12.398091    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:41:12.435658 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:12.435637    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:41:12.451872 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:12.451851    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:41:15.239814 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:15.239785    4464 scope.go:122] "RemoveContainer" containerID="5c708e26d80fb3ff89bf39c0310fc469e06081f6ab814b66e54ec6f4ab586c45"
Nov 02 13:41:15.248350 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:15.248333    4464 scope.go:122] "RemoveContainer" containerID="61496b44e0f213ed7613a6acbc80fa91a02f851ffdeb91bf92d06b80878f00be"
Nov 02 13:41:15.256345 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:15.256326    4464 scope.go:122] "RemoveContainer" containerID="d1827e2baa3e3e1449753357e690c9c1a187af484b8f57add59b4c15eb8700ce"
Nov 02 13:41:22.048079 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:22.048041    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"]
Nov 02 13:41:22.098662 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:22.098629    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-5g5nn"]
Nov 02 13:41:23.580444 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:23.580404    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae727e9d-4854-4596-a194-cfdf6af320de" path="/var/lib/kubelet/pods/ae727e9d-4854-4596-a194-cfdf6af320de/volumes"
Nov 02 13:41:26.907158 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:26.907121    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:41:28.293346 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:41:28.293302    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:41:31.840638 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.840601    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"]
Nov 02 13:41:31.841123 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.841054    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e0eb3d1d-b80e-428e-bd76-9e7d70438152" containerName="revoke-keys"
Nov 02 13:41:31.841123 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.841068    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0eb3d1d-b80e-428e-bd76-9e7d70438152" containerName="revoke-keys"
Nov 02 13:41:31.841123 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.841088    4464 cpu_manager.go:414] "containerMap: removing container" podUID="dff4f721-0c05-44d9-acff-9d121302d980" containerName="istio-proxy"
Nov 02 13:41:31.841123 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.841095    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff4f721-0c05-44d9-acff-9d121302d980" containerName="istio-proxy"
Nov 02 13:41:31.841123 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.841116    4464 cpu_manager.go:414] "containerMap: removing container" podUID="dff4f721-0c05-44d9-acff-9d121302d980" containerName="istio-proxy"
Nov 02 13:41:31.841123 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.841121    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff4f721-0c05-44d9-acff-9d121302d980" containerName="istio-proxy"
Nov 02 13:41:31.841327 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.841180    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0eb3d1d-b80e-428e-bd76-9e7d70438152" containerName="revoke-keys"
Nov 02 13:41:31.841327 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.841191    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff4f721-0c05-44d9-acff-9d121302d980" containerName="istio-proxy"
Nov 02 13:41:31.845910 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.845893    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"
Nov 02 13:41:31.946669 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.946630    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qxpx5\" (UID: \"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"
Nov 02 13:41:31.946860 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.946697    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qxpx5\" (UID: \"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"
Nov 02 13:41:31.946860 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.946772    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d6ph\" (UniqueName: \"kubernetes.io/projected/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-kube-api-access-7d6ph\") pod \"maas-api-revoke-sub-d576e68b-qxpx5\" (UID: \"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"
Nov 02 13:41:31.952096 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.952079    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:41:31.952268 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.952251    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:41:31.952342 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.952282    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:41:31.952342 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.952288    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:41:31.954235 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:31.954217    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"]
Nov 02 13:41:32.047862 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:32.047839    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qxpx5\" (UID: \"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"
Nov 02 13:41:32.048025 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:32.047889    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qxpx5\" (UID: \"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"
Nov 02 13:41:32.048025 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:32.047993    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d6ph\" (UniqueName: \"kubernetes.io/projected/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-kube-api-access-7d6ph\") pod \"maas-api-revoke-sub-d576e68b-qxpx5\" (UID: \"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"
Nov 02 13:41:32.048478 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:32.048456    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qxpx5\" (UID: \"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"
Nov 02 13:41:32.050190 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:32.050173    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qxpx5\" (UID: \"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"
Nov 02 13:41:32.109972 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:32.109912    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d6ph\" (UniqueName: \"kubernetes.io/projected/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-kube-api-access-7d6ph\") pod \"maas-api-revoke-sub-d576e68b-qxpx5\" (UID: \"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"
Nov 02 13:41:32.159048 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:32.159009    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"
Nov 02 13:41:32.529662 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:32.529635    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"]
Nov 02 13:41:32.531458 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:41:32.531431    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fed2b1c_0110_47d3_80a0_9ba9c71e3ee1.slice/crio-7ba46b890994f507b824c378f167e904ada47ef91cfabff9c7339dc64a1d01f4 WatchSource:0}: Error finding container 7ba46b890994f507b824c378f167e904ada47ef91cfabff9c7339dc64a1d01f4: Status 404 returned error can't find the container with id 7ba46b890994f507b824c378f167e904ada47ef91cfabff9c7339dc64a1d01f4
Nov 02 13:41:32.562436 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:32.562405    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5" event={"ID":"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1","Type":"ContainerStarted","Data":"7ba46b890994f507b824c378f167e904ada47ef91cfabff9c7339dc64a1d01f4"}
Nov 02 13:41:33.568991 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:33.568939    4464 generic.go:356] "Generic (PLEG): container finished" podID="2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1" containerID="ffb137ed71b9f8914bd55acd75c94eb7edae83bf65b92d16a98b00cf0afc4f81" exitCode=0
Nov 02 13:41:33.569424 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:33.569027    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5" event={"ID":"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1","Type":"ContainerDied","Data":"ffb137ed71b9f8914bd55acd75c94eb7edae83bf65b92d16a98b00cf0afc4f81"}
Nov 02 13:41:34.708896 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:34.708874    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"
Nov 02 13:41:34.772198 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:34.772172    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-kube-api-access-7d6ph\" (UniqueName: \"kubernetes.io/projected/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-kube-api-access-7d6ph\") pod \"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1\" (UID: \"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1\") "
Nov 02 13:41:34.772373 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:34.772242    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-maas-api-openshift-ca\") pod \"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1\" (UID: \"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1\") "
Nov 02 13:41:34.772373 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:34.772332    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-maas-api-xks-ca\") pod \"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1\" (UID: \"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1\") "
Nov 02 13:41:34.772613 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:34.772589    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-maas-api-openshift-ca" pod "2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1" (UID: "2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:41:34.774168 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:34.774148    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-kube-api-access-7d6ph" pod "2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1" (UID: "2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1"). InnerVolumeSpecName "kube-api-access-7d6ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:41:34.774300 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:34.774283    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-maas-api-xks-ca" pod "2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1" (UID: "2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:41:34.873080 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:34.873021    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d6ph\" (UniqueName: \"kubernetes.io/projected/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-kube-api-access-7d6ph\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:41:34.873080 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:34.873044    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:41:34.873080 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:34.873055    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:41:35.581242 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:35.581212    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"
Nov 02 13:41:35.581429 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:35.581212    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5" event={"ID":"2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1","Type":"ContainerDied","Data":"7ba46b890994f507b824c378f167e904ada47ef91cfabff9c7339dc64a1d01f4"}
Nov 02 13:41:35.581429 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:35.581344    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ba46b890994f507b824c378f167e904ada47ef91cfabff9c7339dc64a1d01f4"
Nov 02 13:41:44.710338 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:44.710305    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:41:52.926366 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:52.926330    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:41:54.275966 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.275930    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"]
Nov 02 13:41:54.276401 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.276366    4464 cpu_manager.go:414] "containerMap: removing container" podUID="2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1" containerName="revoke-keys"
Nov 02 13:41:54.276401 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.276378    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1" containerName="revoke-keys"
Nov 02 13:41:54.276496 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.276464    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff4f721-0c05-44d9-acff-9d121302d980" containerName="istio-proxy"
Nov 02 13:41:54.276496 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.276475    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1" containerName="revoke-keys"
Nov 02 13:41:54.279530 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.279503    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"
Nov 02 13:41:54.335317 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.335286    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:41:54.335462 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.335435    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:41:54.335535 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.335520    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:41:54.365421 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.365397    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79eec0d-4777-41e5-8a43-92549029f6fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-79qnn\" (UID: \"f79eec0d-4777-41e5-8a43-92549029f6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"
Nov 02 13:41:54.365557 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.365513    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79eec0d-4777-41e5-8a43-92549029f6fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-79qnn\" (UID: \"f79eec0d-4777-41e5-8a43-92549029f6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"
Nov 02 13:41:54.365606 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.365565    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm6bj\" (UniqueName: \"kubernetes.io/projected/f79eec0d-4777-41e5-8a43-92549029f6fc-kube-api-access-hm6bj\") pod \"maas-api-revoke-sub-d2bc93bc-79qnn\" (UID: \"f79eec0d-4777-41e5-8a43-92549029f6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"
Nov 02 13:41:54.385815 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.385797    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:41:54.387838 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.387820    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"]
Nov 02 13:41:54.466947 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.466920    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79eec0d-4777-41e5-8a43-92549029f6fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-79qnn\" (UID: \"f79eec0d-4777-41e5-8a43-92549029f6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"
Nov 02 13:41:54.467099 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.466963    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm6bj\" (UniqueName: \"kubernetes.io/projected/f79eec0d-4777-41e5-8a43-92549029f6fc-kube-api-access-hm6bj\") pod \"maas-api-revoke-sub-d2bc93bc-79qnn\" (UID: \"f79eec0d-4777-41e5-8a43-92549029f6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"
Nov 02 13:41:54.467099 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.467036    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79eec0d-4777-41e5-8a43-92549029f6fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-79qnn\" (UID: \"f79eec0d-4777-41e5-8a43-92549029f6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"
Nov 02 13:41:54.467665 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.467646    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79eec0d-4777-41e5-8a43-92549029f6fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-79qnn\" (UID: \"f79eec0d-4777-41e5-8a43-92549029f6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"
Nov 02 13:41:54.469239 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.469220    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79eec0d-4777-41e5-8a43-92549029f6fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-79qnn\" (UID: \"f79eec0d-4777-41e5-8a43-92549029f6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"
Nov 02 13:41:54.528879 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.528828    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm6bj\" (UniqueName: \"kubernetes.io/projected/f79eec0d-4777-41e5-8a43-92549029f6fc-kube-api-access-hm6bj\") pod \"maas-api-revoke-sub-d2bc93bc-79qnn\" (UID: \"f79eec0d-4777-41e5-8a43-92549029f6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"
Nov 02 13:41:54.592061 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.592043    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"
Nov 02 13:41:54.960271 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:54.960241    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"]
Nov 02 13:41:54.961777 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:41:54.961749    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79eec0d_4777_41e5_8a43_92549029f6fc.slice/crio-732bec6288e716d9b46d7a6254719e9eec4cba298ccd71b81ea2438fa7d01b6d WatchSource:0}: Error finding container 732bec6288e716d9b46d7a6254719e9eec4cba298ccd71b81ea2438fa7d01b6d: Status 404 returned error can't find the container with id 732bec6288e716d9b46d7a6254719e9eec4cba298ccd71b81ea2438fa7d01b6d
Nov 02 13:41:55.685653 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:55.685568    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn" event={"ID":"f79eec0d-4777-41e5-8a43-92549029f6fc","Type":"ContainerStarted","Data":"14654ad1be1f5b0b7fc614e5c0a3991ef062b9e7d2fbcc03ddcd1c7625acd295"}
Nov 02 13:41:55.685653 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:55.685628    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn" event={"ID":"f79eec0d-4777-41e5-8a43-92549029f6fc","Type":"ContainerStarted","Data":"732bec6288e716d9b46d7a6254719e9eec4cba298ccd71b81ea2438fa7d01b6d"}
Nov 02 13:41:55.804550 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:55.804499    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn" podStartSLOduration=1.804485261 podStartE2EDuration="1.804485261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:41:55.80317044 +0000 UTC m=+6344.366878847" watchObservedRunningTime="2026-11-02 13:41:55.804485261 +0000 UTC m=+6344.368193695"
Nov 02 13:41:56.691636 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:56.691600    4464 generic.go:356] "Generic (PLEG): container finished" podID="f79eec0d-4777-41e5-8a43-92549029f6fc" containerID="14654ad1be1f5b0b7fc614e5c0a3991ef062b9e7d2fbcc03ddcd1c7625acd295" exitCode=0
Nov 02 13:41:56.692076 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:56.691677    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn" event={"ID":"f79eec0d-4777-41e5-8a43-92549029f6fc","Type":"ContainerDied","Data":"14654ad1be1f5b0b7fc614e5c0a3991ef062b9e7d2fbcc03ddcd1c7625acd295"}
Nov 02 13:41:57.833538 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:57.833502    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"
Nov 02 13:41:57.895931 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:57.895907    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f79eec0d-4777-41e5-8a43-92549029f6fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79eec0d-4777-41e5-8a43-92549029f6fc-maas-api-openshift-ca\") pod \"f79eec0d-4777-41e5-8a43-92549029f6fc\" (UID: \"f79eec0d-4777-41e5-8a43-92549029f6fc\") "
Nov 02 13:41:57.896062 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:57.895989    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79eec0d-4777-41e5-8a43-92549029f6fc-kube-api-access-hm6bj\" (UniqueName: \"kubernetes.io/projected/f79eec0d-4777-41e5-8a43-92549029f6fc-kube-api-access-hm6bj\") pod \"f79eec0d-4777-41e5-8a43-92549029f6fc\" (UID: \"f79eec0d-4777-41e5-8a43-92549029f6fc\") "
Nov 02 13:41:57.896062 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:57.896030    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f79eec0d-4777-41e5-8a43-92549029f6fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79eec0d-4777-41e5-8a43-92549029f6fc-maas-api-xks-ca\") pod \"f79eec0d-4777-41e5-8a43-92549029f6fc\" (UID: \"f79eec0d-4777-41e5-8a43-92549029f6fc\") "
Nov 02 13:41:57.896269 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:57.896247    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f79eec0d-4777-41e5-8a43-92549029f6fc-maas-api-openshift-ca" pod "f79eec0d-4777-41e5-8a43-92549029f6fc" (UID: "f79eec0d-4777-41e5-8a43-92549029f6fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:41:57.896331 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:57.896318    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79eec0d-4777-41e5-8a43-92549029f6fc-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:41:57.897908 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:57.897887    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79eec0d-4777-41e5-8a43-92549029f6fc-kube-api-access-hm6bj" pod "f79eec0d-4777-41e5-8a43-92549029f6fc" (UID: "f79eec0d-4777-41e5-8a43-92549029f6fc"). InnerVolumeSpecName "kube-api-access-hm6bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:41:57.897989 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:57.897953    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f79eec0d-4777-41e5-8a43-92549029f6fc-maas-api-xks-ca" pod "f79eec0d-4777-41e5-8a43-92549029f6fc" (UID: "f79eec0d-4777-41e5-8a43-92549029f6fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:41:57.997603 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:57.997577    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79eec0d-4777-41e5-8a43-92549029f6fc-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:41:57.997603 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:57.997600    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm6bj\" (UniqueName: \"kubernetes.io/projected/f79eec0d-4777-41e5-8a43-92549029f6fc-kube-api-access-hm6bj\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:41:58.291840 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:41:58.291754    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:41:58.702397 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:58.702315    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"
Nov 02 13:41:58.702397 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:58.702362    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn" event={"ID":"f79eec0d-4777-41e5-8a43-92549029f6fc","Type":"ContainerDied","Data":"732bec6288e716d9b46d7a6254719e9eec4cba298ccd71b81ea2438fa7d01b6d"}
Nov 02 13:41:58.702587 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:41:58.702417    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="732bec6288e716d9b46d7a6254719e9eec4cba298ccd71b81ea2438fa7d01b6d"
Nov 02 13:42:01.704350 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:01.704314    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:42:15.286358 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:15.286322    4464 scope.go:122] "RemoveContainer" containerID="1ad15d858f3900ea1ec630f47a3fea1175a657601ed0b03b4c576f2aa645b769"
Nov 02 13:42:15.295191 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:15.295171    4464 scope.go:122] "RemoveContainer" containerID="17e41261cdb2566a4ed0c2a7fb03458afa0ff3d8f205befdca4434e62ebafaed"
Nov 02 13:42:17.774529 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:17.774499    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:42:20.786997 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:20.786679    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"]
Nov 02 13:42:20.803472 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:20.803440    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f79eec0d-4777-41e5-8a43-92549029f6fc" containerName="revoke-keys"
Nov 02 13:42:20.803696 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:20.803676    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f79eec0d-4777-41e5-8a43-92549029f6fc" containerName="revoke-keys"
Nov 02 13:42:20.803995 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:20.803959    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f79eec0d-4777-41e5-8a43-92549029f6fc" containerName="revoke-keys"
Nov 02 13:42:20.813238 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:20.813217    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"
Nov 02 13:42:20.849875 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:20.849847    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"]
Nov 02 13:42:20.898270 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:20.898242    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:42:20.898621 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:20.898602    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:42:20.898861 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:20.898845    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:42:20.910995 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:20.899588    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:42:20.927851 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:20.927821    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ea64b7-vhdtg\" (UID: \"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"
Nov 02 13:42:20.928158 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:20.928134    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ea64b7-vhdtg\" (UID: \"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"
Nov 02 13:42:20.928406 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:20.928387    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqg44\" (UniqueName: \"kubernetes.io/projected/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-kube-api-access-rqg44\") pod \"maas-api-revoke-sub-08ea64b7-vhdtg\" (UID: \"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"
Nov 02 13:42:21.029626 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:21.029587    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ea64b7-vhdtg\" (UID: \"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"
Nov 02 13:42:21.029969 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:21.029947    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqg44\" (UniqueName: \"kubernetes.io/projected/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-kube-api-access-rqg44\") pod \"maas-api-revoke-sub-08ea64b7-vhdtg\" (UID: \"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"
Nov 02 13:42:21.030182 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:21.030153    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ea64b7-vhdtg\" (UID: \"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"
Nov 02 13:42:21.032191 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:21.032171    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ea64b7-vhdtg\" (UID: \"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"
Nov 02 13:42:21.033449 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:21.033429    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ea64b7-vhdtg\" (UID: \"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"
Nov 02 13:42:21.096733 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:21.096640    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqg44\" (UniqueName: \"kubernetes.io/projected/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-kube-api-access-rqg44\") pod \"maas-api-revoke-sub-08ea64b7-vhdtg\" (UID: \"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"
Nov 02 13:42:21.146742 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:21.146717    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"
Nov 02 13:42:21.646557 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:21.646529    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"]
Nov 02 13:42:21.846970 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:21.846936    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg" event={"ID":"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad","Type":"ContainerStarted","Data":"07e200a8220cd69ce490509301ff3609b98f733df3fcaa0ac7136f674a460583"}
Nov 02 13:42:22.853016 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:22.852917    4464 generic.go:356] "Generic (PLEG): container finished" podID="8e4e3bae-2348-4800-8c2c-948f7ba8d4ad" containerID="39e963c7c239e461b23c35f0984484930ebf04f8559e2c2f3e91761a167786d4" exitCode=0
Nov 02 13:42:22.853362 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:22.853004    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg" event={"ID":"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad","Type":"ContainerDied","Data":"39e963c7c239e461b23c35f0984484930ebf04f8559e2c2f3e91761a167786d4"}
Nov 02 13:42:23.990243 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:23.990216    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"
Nov 02 13:42:24.069732 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:24.069702    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-kube-api-access-rqg44\" (UniqueName: \"kubernetes.io/projected/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-kube-api-access-rqg44\") pod \"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad\" (UID: \"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad\") "
Nov 02 13:42:24.069890 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:24.069776    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-maas-api-xks-ca\") pod \"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad\" (UID: \"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad\") "
Nov 02 13:42:24.069890 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:24.069813    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-maas-api-openshift-ca\") pod \"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad\" (UID: \"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad\") "
Nov 02 13:42:24.070260 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:24.070231    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-maas-api-openshift-ca" pod "8e4e3bae-2348-4800-8c2c-948f7ba8d4ad" (UID: "8e4e3bae-2348-4800-8c2c-948f7ba8d4ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:42:24.071869 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:24.071847    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-maas-api-xks-ca" pod "8e4e3bae-2348-4800-8c2c-948f7ba8d4ad" (UID: "8e4e3bae-2348-4800-8c2c-948f7ba8d4ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:42:24.072004 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:24.071847    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-kube-api-access-rqg44" pod "8e4e3bae-2348-4800-8c2c-948f7ba8d4ad" (UID: "8e4e3bae-2348-4800-8c2c-948f7ba8d4ad"). InnerVolumeSpecName "kube-api-access-rqg44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:42:24.171007 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:24.170899    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqg44\" (UniqueName: \"kubernetes.io/projected/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-kube-api-access-rqg44\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:42:24.171007 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:24.170930    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:42:24.171007 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:24.170940    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:42:24.863622 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:24.863594    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"
Nov 02 13:42:24.863796 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:24.863620    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg" event={"ID":"8e4e3bae-2348-4800-8c2c-948f7ba8d4ad","Type":"ContainerDied","Data":"07e200a8220cd69ce490509301ff3609b98f733df3fcaa0ac7136f674a460583"}
Nov 02 13:42:24.863796 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:24.863676    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07e200a8220cd69ce490509301ff3609b98f733df3fcaa0ac7136f674a460583"
Nov 02 13:42:25.987292 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:42:25.987259    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:42:28.291019 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:42:28.290965    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:42:58.291820 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:42:58.291786    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:43:15.324207 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:43:15.324175    4464 scope.go:122] "RemoveContainer" containerID="c9ad480a6a96b22e5a75a195cfbcf1fef0f622b759f9f24e129b1ea0e7d1341a"
Nov 02 13:43:15.332865 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:43:15.332845    4464 scope.go:122] "RemoveContainer" containerID="582c2083abd3959b1a020862a65ce5447c24bcdc8b9489d03d861fa4616ce875"
Nov 02 13:43:16.052086 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:43:16.052047    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"]
Nov 02 13:43:16.102273 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:43:16.102242    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02824528-cq5bd"]
Nov 02 13:43:17.580422 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:43:17.580389    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46" path="/var/lib/kubelet/pods/eb46a194-e8dc-4f61-9d33-0a6ca0fd7d46/volumes"
Nov 02 13:43:28.292268 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:43:28.292186    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:43:58.291949 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:43:58.291907    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:44:15.358235 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:44:15.358202    4464 scope.go:122] "RemoveContainer" containerID="bdc1bf9e6e3b56021b5f620429848b3b17ddf737d217196ff7946b20b5d82406"
Nov 02 13:44:28.291249 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:44:28.291205    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:44:46.051261 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:44:46.051230    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"]
Nov 02 13:44:46.101513 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:44:46.101486    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wgfxn"]
Nov 02 13:44:47.579766 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:44:47.579714    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0eb3d1d-b80e-428e-bd76-9e7d70438152" path="/var/lib/kubelet/pods/e0eb3d1d-b80e-428e-bd76-9e7d70438152/volumes"
Nov 02 13:44:58.290610 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:44:58.290530    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:45:00.176888 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:00.176856    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb"]
Nov 02 13:45:00.177326 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:00.177313    4464 cpu_manager.go:414] "containerMap: removing container" podUID="8e4e3bae-2348-4800-8c2c-948f7ba8d4ad" containerName="revoke-keys"
Nov 02 13:45:00.177374 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:00.177327    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4e3bae-2348-4800-8c2c-948f7ba8d4ad" containerName="revoke-keys"
Nov 02 13:45:00.177410 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:00.177397    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4e3bae-2348-4800-8c2c-948f7ba8d4ad" containerName="revoke-keys"
Nov 02 13:45:00.180703 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:00.180687    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb"
Nov 02 13:45:00.236203 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:00.236177    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:45:00.261733 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:00.261706    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/078c5741-419e-4f63-8583-6271236884cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893785-dj4vb\" (UID: \"078c5741-419e-4f63-8583-6271236884cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb"
Nov 02 13:45:00.261874 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:00.261787    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n98d9\" (UniqueName: \"kubernetes.io/projected/078c5741-419e-4f63-8583-6271236884cf-kube-api-access-n98d9\") pod \"maas-api-key-cleanup-29893785-dj4vb\" (UID: \"078c5741-419e-4f63-8583-6271236884cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb"
Nov 02 13:45:00.284805 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:00.284778    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:45:00.284805 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:00.284790    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:45:00.287070 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:00.287052    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb"]
Nov 02 13:45:00.362394 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:00.362362    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/078c5741-419e-4f63-8583-6271236884cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893785-dj4vb\" (UID: \"078c5741-419e-4f63-8583-6271236884cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb"
Nov 02 13:45:00.362588 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:00.362431    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n98d9\" (UniqueName: \"kubernetes.io/projected/078c5741-419e-4f63-8583-6271236884cf-kube-api-access-n98d9\") pod \"maas-api-key-cleanup-29893785-dj4vb\" (UID: \"078c5741-419e-4f63-8583-6271236884cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb"
Nov 02 13:45:00.363044 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:00.363020    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/078c5741-419e-4f63-8583-6271236884cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893785-dj4vb\" (UID: \"078c5741-419e-4f63-8583-6271236884cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb"
Nov 02 13:45:00.424662 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:00.424639    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n98d9\" (UniqueName: \"kubernetes.io/projected/078c5741-419e-4f63-8583-6271236884cf-kube-api-access-n98d9\") pod \"maas-api-key-cleanup-29893785-dj4vb\" (UID: \"078c5741-419e-4f63-8583-6271236884cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb"
Nov 02 13:45:00.493415 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:00.493394    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb"
Nov 02 13:45:00.870027 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:00.870002    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb"]
Nov 02 13:45:00.871694 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:45:00.871666    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078c5741_419e_4f63_8583_6271236884cf.slice/crio-c02d1ec0c96ef6f949f264ce084e64b0463bbff38fa7ec70d9cd25d3a2b2247f WatchSource:0}: Error finding container c02d1ec0c96ef6f949f264ce084e64b0463bbff38fa7ec70d9cd25d3a2b2247f: Status 404 returned error can't find the container with id c02d1ec0c96ef6f949f264ce084e64b0463bbff38fa7ec70d9cd25d3a2b2247f
Nov 02 13:45:01.648501 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:01.648470    4464 generic.go:356] "Generic (PLEG): container finished" podID="078c5741-419e-4f63-8583-6271236884cf" containerID="82a286e3b1aaebfbc748bc1795e677798b3bb350ec957921df1e055107a68e87" exitCode=60
Nov 02 13:45:01.648948 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:01.648539    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb" event={"ID":"078c5741-419e-4f63-8583-6271236884cf","Type":"ContainerDied","Data":"82a286e3b1aaebfbc748bc1795e677798b3bb350ec957921df1e055107a68e87"}
Nov 02 13:45:01.648948 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:01.648585    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb" event={"ID":"078c5741-419e-4f63-8583-6271236884cf","Type":"ContainerStarted","Data":"c02d1ec0c96ef6f949f264ce084e64b0463bbff38fa7ec70d9cd25d3a2b2247f"}
Nov 02 13:45:01.648948 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:01.648826    4464 scope.go:122] "RemoveContainer" containerID="82a286e3b1aaebfbc748bc1795e677798b3bb350ec957921df1e055107a68e87"
Nov 02 13:45:02.659125 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:02.659039    4464 generic.go:356] "Generic (PLEG): container finished" podID="078c5741-419e-4f63-8583-6271236884cf" containerID="98810f4cf22d8e063f660f1b4745fdbea3c1324c69cb8b78091034237de3b7ab" exitCode=60
Nov 02 13:45:02.659125 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:02.659084    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb" event={"ID":"078c5741-419e-4f63-8583-6271236884cf","Type":"ContainerDied","Data":"98810f4cf22d8e063f660f1b4745fdbea3c1324c69cb8b78091034237de3b7ab"}
Nov 02 13:45:02.659626 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:02.659177    4464 scope.go:122] "RemoveContainer" containerID="82a286e3b1aaebfbc748bc1795e677798b3bb350ec957921df1e055107a68e87"
Nov 02 13:45:02.660043 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:02.660019    4464 scope.go:122] "RemoveContainer" containerID="98810f4cf22d8e063f660f1b4745fdbea3c1324c69cb8b78091034237de3b7ab"
Nov 02 13:45:02.660668 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:45:02.660436    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893785-dj4vb_odh-ai-gateway-infra(078c5741-419e-4f63-8583-6271236884cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb" podUID="078c5741-419e-4f63-8583-6271236884cf"
Nov 02 13:45:03.665851 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:03.665816    4464 scope.go:122] "RemoveContainer" containerID="98810f4cf22d8e063f660f1b4745fdbea3c1324c69cb8b78091034237de3b7ab"
Nov 02 13:45:03.666262 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:45:03.666075    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893785-dj4vb_odh-ai-gateway-infra(078c5741-419e-4f63-8583-6271236884cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb" podUID="078c5741-419e-4f63-8583-6271236884cf"
Nov 02 13:45:16.575192 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:16.575160    4464 scope.go:122] "RemoveContainer" containerID="98810f4cf22d8e063f660f1b4745fdbea3c1324c69cb8b78091034237de3b7ab"
Nov 02 13:45:17.734218 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:17.734184    4464 generic.go:356] "Generic (PLEG): container finished" podID="078c5741-419e-4f63-8583-6271236884cf" containerID="bbce4e32064ced588ea9ccb813b9f1d5e79403685eb097458a0f3f66a6450ae6" exitCode=60
Nov 02 13:45:17.734768 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:17.734258    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb" event={"ID":"078c5741-419e-4f63-8583-6271236884cf","Type":"ContainerDied","Data":"bbce4e32064ced588ea9ccb813b9f1d5e79403685eb097458a0f3f66a6450ae6"}
Nov 02 13:45:17.734768 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:17.734324    4464 scope.go:122] "RemoveContainer" containerID="98810f4cf22d8e063f660f1b4745fdbea3c1324c69cb8b78091034237de3b7ab"
Nov 02 13:45:17.734768 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:17.734682    4464 scope.go:122] "RemoveContainer" containerID="bbce4e32064ced588ea9ccb813b9f1d5e79403685eb097458a0f3f66a6450ae6"
Nov 02 13:45:17.734970 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:45:17.734946    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893785-dj4vb_odh-ai-gateway-infra(078c5741-419e-4f63-8583-6271236884cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb" podUID="078c5741-419e-4f63-8583-6271236884cf"
Nov 02 13:45:18.862425 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:18.862390    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb"]
Nov 02 13:45:19.007611 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:19.007582    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb"
Nov 02 13:45:19.142762 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:19.142679    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/078c5741-419e-4f63-8583-6271236884cf-kube-api-access-n98d9\" (UniqueName: \"kubernetes.io/projected/078c5741-419e-4f63-8583-6271236884cf-kube-api-access-n98d9\") pod \"078c5741-419e-4f63-8583-6271236884cf\" (UID: \"078c5741-419e-4f63-8583-6271236884cf\") "
Nov 02 13:45:19.142940 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:19.142798    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/078c5741-419e-4f63-8583-6271236884cf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/078c5741-419e-4f63-8583-6271236884cf-maas-api-service-ca\") pod \"078c5741-419e-4f63-8583-6271236884cf\" (UID: \"078c5741-419e-4f63-8583-6271236884cf\") "
Nov 02 13:45:19.143219 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:19.143196    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078c5741-419e-4f63-8583-6271236884cf-maas-api-service-ca" pod "078c5741-419e-4f63-8583-6271236884cf" (UID: "078c5741-419e-4f63-8583-6271236884cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:45:19.144931 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:19.144900    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078c5741-419e-4f63-8583-6271236884cf-kube-api-access-n98d9" pod "078c5741-419e-4f63-8583-6271236884cf" (UID: "078c5741-419e-4f63-8583-6271236884cf"). InnerVolumeSpecName "kube-api-access-n98d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:45:19.244411 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:19.244376    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n98d9\" (UniqueName: \"kubernetes.io/projected/078c5741-419e-4f63-8583-6271236884cf-kube-api-access-n98d9\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:45:19.244411 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:19.244407    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/078c5741-419e-4f63-8583-6271236884cf-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:45:19.747555 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:19.747519    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb"
Nov 02 13:45:19.747555 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:19.747534    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb" event={"ID":"078c5741-419e-4f63-8583-6271236884cf","Type":"ContainerDied","Data":"c02d1ec0c96ef6f949f264ce084e64b0463bbff38fa7ec70d9cd25d3a2b2247f"}
Nov 02 13:45:19.747812 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:19.747593    4464 scope.go:122] "RemoveContainer" containerID="bbce4e32064ced588ea9ccb813b9f1d5e79403685eb097458a0f3f66a6450ae6"
Nov 02 13:45:19.914269 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:19.914185    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb"]
Nov 02 13:45:19.964739 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:19.964707    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893785-dj4vb"]
Nov 02 13:45:21.580578 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:45:21.580546    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="078c5741-419e-4f63-8583-6271236884cf" path="/var/lib/kubelet/pods/078c5741-419e-4f63-8583-6271236884cf/volumes"
Nov 02 13:45:28.291760 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:45:28.291718    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:45:58.292083 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:45:58.292049    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:46:12.418116 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:46:12.418085    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:46:12.435365 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:46:12.435340    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:46:12.470571 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:46:12.470544    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:46:12.487162 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:46:12.487131    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:46:15.388711 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:46:15.388679    4464 scope.go:122] "RemoveContainer" containerID="60de7e10367d430ffc04cefbf495227437002d740e6e56b8aedf65dd28124a65"
Nov 02 13:46:15.397351 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:46:15.397331    4464 scope.go:122] "RemoveContainer" containerID="378f58e95870a9a0484ef33313fa06ac8bbc560a53cf0b797f1f4cdb60e78358"
Nov 02 13:46:28.291184 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:46:28.291099    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:46:35.056991 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:46:35.056934    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"]
Nov 02 13:46:35.106971 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:46:35.106935    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qxpx5"]
Nov 02 13:46:35.582040 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:46:35.581995    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1" path="/var/lib/kubelet/pods/2fed2b1c-0110-47d3-80a0-9ba9c71e3ee1/volumes"
Nov 02 13:46:58.047397 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:46:58.047359    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"]
Nov 02 13:46:58.097616 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:46:58.097585    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-79qnn"]
Nov 02 13:46:58.292145 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:46:58.292111    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:46:59.580541 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:46:59.580505    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f79eec0d-4777-41e5-8a43-92549029f6fc" path="/var/lib/kubelet/pods/f79eec0d-4777-41e5-8a43-92549029f6fc/volumes"
Nov 02 13:47:25.049536 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:25.049491    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"]
Nov 02 13:47:25.100198 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:25.100159    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ea64b7-vhdtg"]
Nov 02 13:47:25.581027 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:25.580994    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4e3bae-2348-4800-8c2c-948f7ba8d4ad" path="/var/lib/kubelet/pods/8e4e3bae-2348-4800-8c2c-948f7ba8d4ad/volumes"
Nov 02 13:47:28.291529 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:47:28.291490    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:47:51.937856 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:51.937817    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"]
Nov 02 13:47:51.938418 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:51.938058    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7" podUID="02c134c2-e2f8-4a16-a557-65786f86217c" containerName="istio-proxy" containerID="cri-o://7a94b036398001bc4545922a1913a86dbc8ffecd61a4a6bb5be77b6db2f448fc" gracePeriod=30
Nov 02 13:47:51.988679 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:51.988645    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"]
Nov 02 13:47:51.988873 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:51.988846    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls" podUID="85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" containerName="istio-proxy" containerID="cri-o://b2afcfa3c234579a7156db6bffb43032ffdfdbf6a25513bfc48e923ded28e194" gracePeriod=30
Nov 02 13:47:57.239470 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.239443    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:47:57.265243 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.265220    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:47:57.354857 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.354819    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-workload-certs\") pod \"02c134c2-e2f8-4a16-a557-65786f86217c\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") "
Nov 02 13:47:57.355059 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.354889    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-istio-envoy\") pod \"02c134c2-e2f8-4a16-a557-65786f86217c\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") "
Nov 02 13:47:57.355059 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.354941    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-kube-api-access-kv8tl\" (UniqueName: \"kubernetes.io/projected/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-kube-api-access-kv8tl\") pod \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") "
Nov 02 13:47:57.355059 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.354996    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-podinfo\") pod \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") "
Nov 02 13:47:57.355059 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.355026    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-istio-data\") pod \"02c134c2-e2f8-4a16-a557-65786f86217c\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") "
Nov 02 13:47:57.355059 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.355054    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-data\") pod \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") "
Nov 02 13:47:57.355325 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.355085    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-token\" (UniqueName: \"kubernetes.io/projected/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-token\") pod \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") "
Nov 02 13:47:57.355325 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.355112    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-workload-certs" pod "02c134c2-e2f8-4a16-a557-65786f86217c" (UID: "02c134c2-e2f8-4a16-a557-65786f86217c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:47:57.355325 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.355136    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istiod-ca-cert\") pod \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") "
Nov 02 13:47:57.355325 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.355167    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-workload-certs\") pod \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") "
Nov 02 13:47:57.355325 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.355209    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c134c2-e2f8-4a16-a557-65786f86217c-istio-token\" (UniqueName: \"kubernetes.io/projected/02c134c2-e2f8-4a16-a557-65786f86217c-istio-token\") pod \"02c134c2-e2f8-4a16-a557-65786f86217c\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") "
Nov 02 13:47:57.355325 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.355248    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c134c2-e2f8-4a16-a557-65786f86217c-kube-api-access-zhwp5\" (UniqueName: \"kubernetes.io/projected/02c134c2-e2f8-4a16-a557-65786f86217c-kube-api-access-zhwp5\") pod \"02c134c2-e2f8-4a16-a557-65786f86217c\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") "
Nov 02 13:47:57.355675 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.355361    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-istio-data" pod "02c134c2-e2f8-4a16-a557-65786f86217c" (UID: "02c134c2-e2f8-4a16-a557-65786f86217c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:47:57.355675 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.355464    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02c134c2-e2f8-4a16-a557-65786f86217c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c134c2-e2f8-4a16-a557-65786f86217c-istio-podinfo\") pod \"02c134c2-e2f8-4a16-a557-65786f86217c\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") "
Nov 02 13:47:57.355675 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.355511    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-workload-socket\") pod \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") "
Nov 02 13:47:57.355675 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.355575    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-credential-socket\") pod \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") "
Nov 02 13:47:57.355675 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.355615    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-workload-socket\") pod \"02c134c2-e2f8-4a16-a557-65786f86217c\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") "
Nov 02 13:47:57.355675 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.355673    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c134c2-e2f8-4a16-a557-65786f86217c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c134c2-e2f8-4a16-a557-65786f86217c-istiod-ca-cert\") pod \"02c134c2-e2f8-4a16-a557-65786f86217c\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") "
Nov 02 13:47:57.356043 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.355715    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-credential-socket\") pod \"02c134c2-e2f8-4a16-a557-65786f86217c\" (UID: \"02c134c2-e2f8-4a16-a557-65786f86217c\") "
Nov 02 13:47:57.356043 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.355711    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-data" pod "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" (UID: "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:47:57.356043 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.355746    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-envoy\") pod \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\" (UID: \"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921\") "
Nov 02 13:47:57.356229 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.356134    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.356229 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.356152    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.356229 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.356166    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.356407 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.356244    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istiod-ca-cert" pod "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" (UID: "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:47:57.356998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.356585    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-workload-socket" pod "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" (UID: "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:47:57.356998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.356626    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c134c2-e2f8-4a16-a557-65786f86217c-istiod-ca-cert" pod "02c134c2-e2f8-4a16-a557-65786f86217c" (UID: "02c134c2-e2f8-4a16-a557-65786f86217c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:47:57.356998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.356656    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-workload-socket" pod "02c134c2-e2f8-4a16-a557-65786f86217c" (UID: "02c134c2-e2f8-4a16-a557-65786f86217c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:47:57.356998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.356728    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-credential-socket" pod "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" (UID: "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:47:57.356998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.356805    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-workload-certs" pod "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" (UID: "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:47:57.356998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.356934    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-credential-socket" pod "02c134c2-e2f8-4a16-a557-65786f86217c" (UID: "02c134c2-e2f8-4a16-a557-65786f86217c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:47:57.357626 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.357600    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-kube-api-access-kv8tl" pod "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" (UID: "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921"). InnerVolumeSpecName "kube-api-access-kv8tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:47:57.358185 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.358160    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c134c2-e2f8-4a16-a557-65786f86217c-istio-token" pod "02c134c2-e2f8-4a16-a557-65786f86217c" (UID: "02c134c2-e2f8-4a16-a557-65786f86217c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:47:57.358491 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.358459    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-envoy" pod "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" (UID: "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:47:57.358491 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.358473    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-istio-envoy" pod "02c134c2-e2f8-4a16-a557-65786f86217c" (UID: "02c134c2-e2f8-4a16-a557-65786f86217c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:47:57.358809 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.358788    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c134c2-e2f8-4a16-a557-65786f86217c-kube-api-access-zhwp5" pod "02c134c2-e2f8-4a16-a557-65786f86217c" (UID: "02c134c2-e2f8-4a16-a557-65786f86217c"). InnerVolumeSpecName "kube-api-access-zhwp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:47:57.359082 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.359064    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-podinfo" pod "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" (UID: "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:47:57.359132 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.359095    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-token" pod "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" (UID: "85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:47:57.359697 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.359677    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02c134c2-e2f8-4a16-a557-65786f86217c-istio-podinfo" pod "02c134c2-e2f8-4a16-a557-65786f86217c" (UID: "02c134c2-e2f8-4a16-a557-65786f86217c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:47:57.456897 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.456856    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.456897 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.456890    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.456897 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.456900    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.457171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.456909    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv8tl\" (UniqueName: \"kubernetes.io/projected/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-kube-api-access-kv8tl\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.457171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.456920    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.457171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.456928    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.457171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.456936    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.457171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.456944    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.457171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.456958    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c134c2-e2f8-4a16-a557-65786f86217c-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.457171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.456970    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhwp5\" (UniqueName: \"kubernetes.io/projected/02c134c2-e2f8-4a16-a557-65786f86217c-kube-api-access-zhwp5\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.457171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.456990    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c134c2-e2f8-4a16-a557-65786f86217c-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.457171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.456999    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.457171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.457009    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.457171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.457017    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c134c2-e2f8-4a16-a557-65786f86217c-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.457171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.457026    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c134c2-e2f8-4a16-a557-65786f86217c-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:47:57.533777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.533694    4464 generic.go:356] "Generic (PLEG): container finished" podID="85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" containerID="b2afcfa3c234579a7156db6bffb43032ffdfdbf6a25513bfc48e923ded28e194" exitCode=0
Nov 02 13:47:57.533777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.533747    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"
Nov 02 13:47:57.534067 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.533782    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls" event={"ID":"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921","Type":"ContainerDied","Data":"b2afcfa3c234579a7156db6bffb43032ffdfdbf6a25513bfc48e923ded28e194"}
Nov 02 13:47:57.534067 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.533867    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls" event={"ID":"85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921","Type":"ContainerDied","Data":"f0382797fb81b7d654bccc109f826459e4c2d2e54843f0533f1979a9463219b7"}
Nov 02 13:47:57.534067 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.533894    4464 scope.go:122] "RemoveContainer" containerID="b2afcfa3c234579a7156db6bffb43032ffdfdbf6a25513bfc48e923ded28e194"
Nov 02 13:47:57.540200 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.540037    4464 generic.go:356] "Generic (PLEG): container finished" podID="02c134c2-e2f8-4a16-a557-65786f86217c" containerID="7a94b036398001bc4545922a1913a86dbc8ffecd61a4a6bb5be77b6db2f448fc" exitCode=0
Nov 02 13:47:57.540200 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.540074    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7" event={"ID":"02c134c2-e2f8-4a16-a557-65786f86217c","Type":"ContainerDied","Data":"7a94b036398001bc4545922a1913a86dbc8ffecd61a4a6bb5be77b6db2f448fc"}
Nov 02 13:47:57.540200 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.540118    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7" event={"ID":"02c134c2-e2f8-4a16-a557-65786f86217c","Type":"ContainerDied","Data":"20183dbb830d4420c77d163349aeff31c0c1f742d3fae797a81e499a11612457"}
Nov 02 13:47:57.540414 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.540241    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"
Nov 02 13:47:57.551460 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.550918    4464 scope.go:122] "RemoveContainer" containerID="b2afcfa3c234579a7156db6bffb43032ffdfdbf6a25513bfc48e923ded28e194"
Nov 02 13:47:57.553666 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:47:57.553635    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2afcfa3c234579a7156db6bffb43032ffdfdbf6a25513bfc48e923ded28e194\": container with ID starting with b2afcfa3c234579a7156db6bffb43032ffdfdbf6a25513bfc48e923ded28e194 not found: ID does not exist" containerID="b2afcfa3c234579a7156db6bffb43032ffdfdbf6a25513bfc48e923ded28e194"
Nov 02 13:47:57.553785 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.553678    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2afcfa3c234579a7156db6bffb43032ffdfdbf6a25513bfc48e923ded28e194"} err="failed to get container status \"b2afcfa3c234579a7156db6bffb43032ffdfdbf6a25513bfc48e923ded28e194\": rpc error: code = NotFound desc = could not find container \"b2afcfa3c234579a7156db6bffb43032ffdfdbf6a25513bfc48e923ded28e194\": container with ID starting with b2afcfa3c234579a7156db6bffb43032ffdfdbf6a25513bfc48e923ded28e194 not found: ID does not exist"
Nov 02 13:47:57.553785 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.553698    4464 scope.go:122] "RemoveContainer" containerID="7a94b036398001bc4545922a1913a86dbc8ffecd61a4a6bb5be77b6db2f448fc"
Nov 02 13:47:57.564465 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.564440    4464 scope.go:122] "RemoveContainer" containerID="7a94b036398001bc4545922a1913a86dbc8ffecd61a4a6bb5be77b6db2f448fc"
Nov 02 13:47:57.564779 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:47:57.564760    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a94b036398001bc4545922a1913a86dbc8ffecd61a4a6bb5be77b6db2f448fc\": container with ID starting with 7a94b036398001bc4545922a1913a86dbc8ffecd61a4a6bb5be77b6db2f448fc not found: ID does not exist" containerID="7a94b036398001bc4545922a1913a86dbc8ffecd61a4a6bb5be77b6db2f448fc"
Nov 02 13:47:57.564842 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.564790    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a94b036398001bc4545922a1913a86dbc8ffecd61a4a6bb5be77b6db2f448fc"} err="failed to get container status \"7a94b036398001bc4545922a1913a86dbc8ffecd61a4a6bb5be77b6db2f448fc\": rpc error: code = NotFound desc = could not find container \"7a94b036398001bc4545922a1913a86dbc8ffecd61a4a6bb5be77b6db2f448fc\": container with ID starting with 7a94b036398001bc4545922a1913a86dbc8ffecd61a4a6bb5be77b6db2f448fc not found: ID does not exist"
Nov 02 13:47:57.703118 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.703088    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"]
Nov 02 13:47:57.707970 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.707941    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-bp4h7"]
Nov 02 13:47:57.819835 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.819752    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"]
Nov 02 13:47:57.827349 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:57.827319    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zfkls"]
Nov 02 13:47:58.291449 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:47:58.291406    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:47:59.580391 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:59.580360    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c134c2-e2f8-4a16-a557-65786f86217c" path="/var/lib/kubelet/pods/02c134c2-e2f8-4a16-a557-65786f86217c/volumes"
Nov 02 13:47:59.580789 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:47:59.580776    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" path="/var/lib/kubelet/pods/85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921/volumes"
Nov 02 13:48:15.433738 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:48:15.433703    4464 scope.go:122] "RemoveContainer" containerID="ffb137ed71b9f8914bd55acd75c94eb7edae83bf65b92d16a98b00cf0afc4f81"
Nov 02 13:48:15.442431 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:48:15.442405    4464 scope.go:122] "RemoveContainer" containerID="14654ad1be1f5b0b7fc614e5c0a3991ef062b9e7d2fbcc03ddcd1c7625acd295"
Nov 02 13:48:28.291199 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:48:28.291160    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:48:58.290075 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:48:58.290041    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:49:15.472021 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:15.471964    4464 scope.go:122] "RemoveContainer" containerID="39e963c7c239e461b23c35f0984484930ebf04f8559e2c2f3e91761a167786d4"
Nov 02 13:49:28.291285 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:49:28.291208    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:49:37.846788 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.846751    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"]
Nov 02 13:49:37.847213 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.847181    4464 cpu_manager.go:414] "containerMap: removing container" podUID="078c5741-419e-4f63-8583-6271236884cf" containerName="cleanup"
Nov 02 13:49:37.847213 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.847193    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078c5741-419e-4f63-8583-6271236884cf" containerName="cleanup"
Nov 02 13:49:37.847213 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.847201    4464 cpu_manager.go:414] "containerMap: removing container" podUID="078c5741-419e-4f63-8583-6271236884cf" containerName="cleanup"
Nov 02 13:49:37.847213 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.847206    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078c5741-419e-4f63-8583-6271236884cf" containerName="cleanup"
Nov 02 13:49:37.847364 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.847224    4464 cpu_manager.go:414] "containerMap: removing container" podUID="85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" containerName="istio-proxy"
Nov 02 13:49:37.847364 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.847230    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" containerName="istio-proxy"
Nov 02 13:49:37.847364 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.847246    4464 cpu_manager.go:414] "containerMap: removing container" podUID="02c134c2-e2f8-4a16-a557-65786f86217c" containerName="istio-proxy"
Nov 02 13:49:37.847364 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.847250    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c134c2-e2f8-4a16-a557-65786f86217c" containerName="istio-proxy"
Nov 02 13:49:37.847364 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.847318    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="078c5741-419e-4f63-8583-6271236884cf" containerName="cleanup"
Nov 02 13:49:37.847364 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.847325    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="078c5741-419e-4f63-8583-6271236884cf" containerName="cleanup"
Nov 02 13:49:37.847364 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.847331    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b4bfae-a5c2-4eec-ae7a-d3fd55c1f921" containerName="istio-proxy"
Nov 02 13:49:37.847364 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.847339    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c134c2-e2f8-4a16-a557-65786f86217c" containerName="istio-proxy"
Nov 02 13:49:37.851636 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.851619    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"
Nov 02 13:49:37.907065 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.907038    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:49:37.909802 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.909785    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf62f22d-f314-4c99-a0f7-6ffd807c5148-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5206625f-cpj4j\" (UID: \"cf62f22d-f314-4c99-a0f7-6ffd807c5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"
Nov 02 13:49:37.909871 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.909832    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf62f22d-f314-4c99-a0f7-6ffd807c5148-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5206625f-cpj4j\" (UID: \"cf62f22d-f314-4c99-a0f7-6ffd807c5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"
Nov 02 13:49:37.909985 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.909957    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bxv6\" (UniqueName: \"kubernetes.io/projected/cf62f22d-f314-4c99-a0f7-6ffd807c5148-kube-api-access-9bxv6\") pod \"maas-api-revoke-sub-5206625f-cpj4j\" (UID: \"cf62f22d-f314-4c99-a0f7-6ffd807c5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"
Nov 02 13:49:37.957352 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.957328    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:49:37.957352 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.957335    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:49:37.957352 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.957352    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:49:37.986075 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:37.986035    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"]
Nov 02 13:49:38.010488 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:38.010455    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bxv6\" (UniqueName: \"kubernetes.io/projected/cf62f22d-f314-4c99-a0f7-6ffd807c5148-kube-api-access-9bxv6\") pod \"maas-api-revoke-sub-5206625f-cpj4j\" (UID: \"cf62f22d-f314-4c99-a0f7-6ffd807c5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"
Nov 02 13:49:38.010700 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:38.010516    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf62f22d-f314-4c99-a0f7-6ffd807c5148-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5206625f-cpj4j\" (UID: \"cf62f22d-f314-4c99-a0f7-6ffd807c5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"
Nov 02 13:49:38.010700 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:38.010580    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf62f22d-f314-4c99-a0f7-6ffd807c5148-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5206625f-cpj4j\" (UID: \"cf62f22d-f314-4c99-a0f7-6ffd807c5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"
Nov 02 13:49:38.011303 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:38.011278    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf62f22d-f314-4c99-a0f7-6ffd807c5148-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5206625f-cpj4j\" (UID: \"cf62f22d-f314-4c99-a0f7-6ffd807c5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"
Nov 02 13:49:38.012849 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:38.012826    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf62f22d-f314-4c99-a0f7-6ffd807c5148-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5206625f-cpj4j\" (UID: \"cf62f22d-f314-4c99-a0f7-6ffd807c5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"
Nov 02 13:49:38.085454 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:38.085425    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bxv6\" (UniqueName: \"kubernetes.io/projected/cf62f22d-f314-4c99-a0f7-6ffd807c5148-kube-api-access-9bxv6\") pod \"maas-api-revoke-sub-5206625f-cpj4j\" (UID: \"cf62f22d-f314-4c99-a0f7-6ffd807c5148\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"
Nov 02 13:49:38.168026 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:38.167942    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"
Nov 02 13:49:38.539628 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:38.539601    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"]
Nov 02 13:49:38.541779 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:49:38.541751    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf62f22d_f314_4c99_a0f7_6ffd807c5148.slice/crio-1c7d420d75caae394d9601d7f9c3a24b7e7758cb3e6287cf1aa41bc1c422fdf7 WatchSource:0}: Error finding container 1c7d420d75caae394d9601d7f9c3a24b7e7758cb3e6287cf1aa41bc1c422fdf7: Status 404 returned error can't find the container with id 1c7d420d75caae394d9601d7f9c3a24b7e7758cb3e6287cf1aa41bc1c422fdf7
Nov 02 13:49:39.007054 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:39.007013    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j" event={"ID":"cf62f22d-f314-4c99-a0f7-6ffd807c5148","Type":"ContainerStarted","Data":"cb0dcda686b670beaf37f2741230751a1894c34e84cdef894778db589c970c90"}
Nov 02 13:49:39.007473 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:39.007070    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j" event={"ID":"cf62f22d-f314-4c99-a0f7-6ffd807c5148","Type":"ContainerStarted","Data":"1c7d420d75caae394d9601d7f9c3a24b7e7758cb3e6287cf1aa41bc1c422fdf7"}
Nov 02 13:49:39.125194 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:39.125098    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j" podStartSLOduration=2.125083445 podStartE2EDuration="2.125083445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:39.124053039 +0000 UTC m=+6807.687761451" watchObservedRunningTime="2026-11-02 13:49:39.125083445 +0000 UTC m=+6807.688791856"
Nov 02 13:49:40.013687 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:40.013655    4464 generic.go:356] "Generic (PLEG): container finished" podID="cf62f22d-f314-4c99-a0f7-6ffd807c5148" containerID="cb0dcda686b670beaf37f2741230751a1894c34e84cdef894778db589c970c90" exitCode=0
Nov 02 13:49:40.014104 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:40.013699    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j" event={"ID":"cf62f22d-f314-4c99-a0f7-6ffd807c5148","Type":"ContainerDied","Data":"cb0dcda686b670beaf37f2741230751a1894c34e84cdef894778db589c970c90"}
Nov 02 13:49:40.118941 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:40.118905    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:49:41.148443 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:41.148419    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"
Nov 02 13:49:41.240252 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:41.240228    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf62f22d-f314-4c99-a0f7-6ffd807c5148-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf62f22d-f314-4c99-a0f7-6ffd807c5148-maas-api-openshift-ca\") pod \"cf62f22d-f314-4c99-a0f7-6ffd807c5148\" (UID: \"cf62f22d-f314-4c99-a0f7-6ffd807c5148\") "
Nov 02 13:49:41.240408 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:41.240291    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf62f22d-f314-4c99-a0f7-6ffd807c5148-kube-api-access-9bxv6\" (UniqueName: \"kubernetes.io/projected/cf62f22d-f314-4c99-a0f7-6ffd807c5148-kube-api-access-9bxv6\") pod \"cf62f22d-f314-4c99-a0f7-6ffd807c5148\" (UID: \"cf62f22d-f314-4c99-a0f7-6ffd807c5148\") "
Nov 02 13:49:41.240408 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:41.240351    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf62f22d-f314-4c99-a0f7-6ffd807c5148-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf62f22d-f314-4c99-a0f7-6ffd807c5148-maas-api-xks-ca\") pod \"cf62f22d-f314-4c99-a0f7-6ffd807c5148\" (UID: \"cf62f22d-f314-4c99-a0f7-6ffd807c5148\") "
Nov 02 13:49:41.240617 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:41.240595    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf62f22d-f314-4c99-a0f7-6ffd807c5148-maas-api-openshift-ca" pod "cf62f22d-f314-4c99-a0f7-6ffd807c5148" (UID: "cf62f22d-f314-4c99-a0f7-6ffd807c5148"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:49:41.242352 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:41.242325    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf62f22d-f314-4c99-a0f7-6ffd807c5148-maas-api-xks-ca" pod "cf62f22d-f314-4c99-a0f7-6ffd807c5148" (UID: "cf62f22d-f314-4c99-a0f7-6ffd807c5148"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:49:41.242453 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:41.242344    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf62f22d-f314-4c99-a0f7-6ffd807c5148-kube-api-access-9bxv6" pod "cf62f22d-f314-4c99-a0f7-6ffd807c5148" (UID: "cf62f22d-f314-4c99-a0f7-6ffd807c5148"). InnerVolumeSpecName "kube-api-access-9bxv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:49:41.341783 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:41.341713    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf62f22d-f314-4c99-a0f7-6ffd807c5148-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:49:41.341783 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:41.341740    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf62f22d-f314-4c99-a0f7-6ffd807c5148-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:49:41.341783 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:41.341752    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bxv6\" (UniqueName: \"kubernetes.io/projected/cf62f22d-f314-4c99-a0f7-6ffd807c5148-kube-api-access-9bxv6\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:49:42.026082 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:42.026043    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"
Nov 02 13:49:42.026293 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:42.026080    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j" event={"ID":"cf62f22d-f314-4c99-a0f7-6ffd807c5148","Type":"ContainerDied","Data":"1c7d420d75caae394d9601d7f9c3a24b7e7758cb3e6287cf1aa41bc1c422fdf7"}
Nov 02 13:49:42.026293 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:49:42.026142    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c7d420d75caae394d9601d7f9c3a24b7e7758cb3e6287cf1aa41bc1c422fdf7"
Nov 02 13:49:58.291103 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:49:58.291061    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:50:08.074541 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.074504    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"]
Nov 02 13:50:08.074951 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.074927    4464 cpu_manager.go:414] "containerMap: removing container" podUID="078c5741-419e-4f63-8583-6271236884cf" containerName="cleanup"
Nov 02 13:50:08.074951 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.074939    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078c5741-419e-4f63-8583-6271236884cf" containerName="cleanup"
Nov 02 13:50:08.075073 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.074960    4464 cpu_manager.go:414] "containerMap: removing container" podUID="cf62f22d-f314-4c99-a0f7-6ffd807c5148" containerName="revoke-keys"
Nov 02 13:50:08.075073 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.074966    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf62f22d-f314-4c99-a0f7-6ffd807c5148" containerName="revoke-keys"
Nov 02 13:50:08.075073 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.075068    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf62f22d-f314-4c99-a0f7-6ffd807c5148" containerName="revoke-keys"
Nov 02 13:50:08.075177 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.075082    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="078c5741-419e-4f63-8583-6271236884cf" containerName="cleanup"
Nov 02 13:50:08.078332 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.078315    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.176733 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.176700    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.176911 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.176740    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.176911 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.176835    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.176911 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.176867    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.176911 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.176891    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.177141 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.176917    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.177141 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.177052    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggnxr\" (UniqueName: \"kubernetes.io/projected/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-kube-api-access-ggnxr\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.177141 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.177084    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.177141 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.177114    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.207306 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.207270    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"]
Nov 02 13:50:08.278301 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.278267    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.278496 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.278345    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.278496 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.278380    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.278496 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.278456    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.278704 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.278499    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.278704 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.278519    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.278704 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.278539    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.278704 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.278590    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggnxr\" (UniqueName: \"kubernetes.io/projected/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-kube-api-access-ggnxr\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.278704 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.278621    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.279016 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.278899    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.279016 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.278998    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.279146 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.279049    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.279252 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.279232    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.279309 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.279266    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.280882 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.280864    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.281139 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.281119    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.340011 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.339940    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.341614 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.341596    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggnxr\" (UniqueName: \"kubernetes.io/projected/6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116-kube-api-access-ggnxr\") pod \"maas-default-gateway-openshift-default-6dcf757899-pk25x\" (UID: \"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.390887 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.390860    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:08.818651 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.818619    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"]
Nov 02 13:50:08.819777 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:50:08.819753    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c163eda_a3a0_4ea5_a8ea_ccc9f1d15116.slice/crio-ad6e6f9cc5ded5090de5c72e94f194116d4ee8b60bc6938b2254623084c30e40 WatchSource:0}: Error finding container ad6e6f9cc5ded5090de5c72e94f194116d4ee8b60bc6938b2254623084c30e40: Status 404 returned error can't find the container with id ad6e6f9cc5ded5090de5c72e94f194116d4ee8b60bc6938b2254623084c30e40
Nov 02 13:50:08.822034 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.821998    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:50:08.822115 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.822077    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:50:08.822115 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:08.822107    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:50:09.164062 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:09.163944    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x" event={"ID":"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116","Type":"ContainerStarted","Data":"8af9f4bf235801665f4fff0b19b47bb18fcf69220f455c030804f56f3e78807f"}
Nov 02 13:50:09.164062 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:09.164016    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x" event={"ID":"6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116","Type":"ContainerStarted","Data":"ad6e6f9cc5ded5090de5c72e94f194116d4ee8b60bc6938b2254623084c30e40"}
Nov 02 13:50:09.391404 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:09.391368    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:09.393003 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:09.392959    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-pk25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body=
Nov 02 13:50:09.393144 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:09.393020    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x" podUID="6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused"
Nov 02 13:50:10.391652 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:10.391619    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-pk25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body=
Nov 02 13:50:10.392093 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:10.391674    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x" podUID="6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused"
Nov 02 13:50:11.392158 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:11.392114    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-pk25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body=
Nov 02 13:50:11.392570 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:11.392167    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x" podUID="6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused"
Nov 02 13:50:12.391654 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:12.391620    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-pk25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body=
Nov 02 13:50:12.391842 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:12.391673    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x" podUID="6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused"
Nov 02 13:50:13.391495 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:13.391453    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-pk25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body=
Nov 02 13:50:13.391911 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:13.391515    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x" podUID="6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused"
Nov 02 13:50:14.391925 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:14.391883    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-pk25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body=
Nov 02 13:50:14.392339 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:14.391940    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x" podUID="6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused"
Nov 02 13:50:15.392067 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:15.392020    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-pk25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body=
Nov 02 13:50:15.392583 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:15.392077    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x" podUID="6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused"
Nov 02 13:50:16.392168 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:16.392133    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-pk25x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body=
Nov 02 13:50:16.392558 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:16.392187    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x" podUID="6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused"
Nov 02 13:50:18.166333 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:18.166301    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:18.166865 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:18.166663    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:18.167530 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:18.167505    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x"
Nov 02 13:50:18.342218 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:50:18.342169    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x" podStartSLOduration=10.342156969 podStartE2EDuration="10.342156969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:09.290940215 +0000 UTC m=+6837.854648625" watchObservedRunningTime="2026-11-02 13:50:18.342156969 +0000 UTC m=+6846.905865377"
Nov 02 13:50:28.292486 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:50:28.292449    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:50:58.290827 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:50:58.290742    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:51:12.450262 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:12.450235    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:51:12.466179 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:12.466152    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:51:12.506553 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:12.506529    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:51:12.522142 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:12.522119    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:51:19.452188 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:19.452148    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"]
Nov 02 13:51:19.456171 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:19.456141    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"
Nov 02 13:51:19.517249 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:19.517218    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"]
Nov 02 13:51:19.544154 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:19.544118    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a14d2b91-407b-4843-af74-67b3298caf27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2266010-d445x\" (UID: \"a14d2b91-407b-4843-af74-67b3298caf27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"
Nov 02 13:51:19.544307 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:19.544228    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a14d2b91-407b-4843-af74-67b3298caf27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2266010-d445x\" (UID: \"a14d2b91-407b-4843-af74-67b3298caf27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"
Nov 02 13:51:19.544307 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:19.544273    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb2xl\" (UniqueName: \"kubernetes.io/projected/a14d2b91-407b-4843-af74-67b3298caf27-kube-api-access-mb2xl\") pod \"maas-api-revoke-sub-b2266010-d445x\" (UID: \"a14d2b91-407b-4843-af74-67b3298caf27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"
Nov 02 13:51:19.565301 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:19.565278    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:51:19.565549 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:19.565533    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:51:19.565628 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:19.565591    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:51:19.565673 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:19.565636    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:51:19.645103 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:19.645080    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a14d2b91-407b-4843-af74-67b3298caf27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2266010-d445x\" (UID: \"a14d2b91-407b-4843-af74-67b3298caf27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"
Nov 02 13:51:19.645245 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:19.645228    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a14d2b91-407b-4843-af74-67b3298caf27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2266010-d445x\" (UID: \"a14d2b91-407b-4843-af74-67b3298caf27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"
Nov 02 13:51:19.645330 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:19.645268    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb2xl\" (UniqueName: \"kubernetes.io/projected/a14d2b91-407b-4843-af74-67b3298caf27-kube-api-access-mb2xl\") pod \"maas-api-revoke-sub-b2266010-d445x\" (UID: \"a14d2b91-407b-4843-af74-67b3298caf27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"
Nov 02 13:51:19.645914 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:19.645891    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a14d2b91-407b-4843-af74-67b3298caf27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2266010-d445x\" (UID: \"a14d2b91-407b-4843-af74-67b3298caf27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"
Nov 02 13:51:19.647539 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:19.647518    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a14d2b91-407b-4843-af74-67b3298caf27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2266010-d445x\" (UID: \"a14d2b91-407b-4843-af74-67b3298caf27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"
Nov 02 13:51:19.707107 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:19.707053    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb2xl\" (UniqueName: \"kubernetes.io/projected/a14d2b91-407b-4843-af74-67b3298caf27-kube-api-access-mb2xl\") pod \"maas-api-revoke-sub-b2266010-d445x\" (UID: \"a14d2b91-407b-4843-af74-67b3298caf27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"
Nov 02 13:51:19.768912 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:19.768890    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"
Nov 02 13:51:20.139555 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:20.139524    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"]
Nov 02 13:51:20.141384 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:51:20.141359    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14d2b91_407b_4843_af74_67b3298caf27.slice/crio-8e91ab85f7782c14c98056893ba09b5fcbc3c60f1d123611c22748251e5dda94 WatchSource:0}: Error finding container 8e91ab85f7782c14c98056893ba09b5fcbc3c60f1d123611c22748251e5dda94: Status 404 returned error can't find the container with id 8e91ab85f7782c14c98056893ba09b5fcbc3c60f1d123611c22748251e5dda94
Nov 02 13:51:20.529703 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:20.529676    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x" event={"ID":"a14d2b91-407b-4843-af74-67b3298caf27","Type":"ContainerStarted","Data":"bd1916845e8e13b3112c21bc277fc8dc2dc3a9e50990891bc2a1c0769665abd4"}
Nov 02 13:51:20.530070 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:20.529724    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x" event={"ID":"a14d2b91-407b-4843-af74-67b3298caf27","Type":"ContainerStarted","Data":"8e91ab85f7782c14c98056893ba09b5fcbc3c60f1d123611c22748251e5dda94"}
Nov 02 13:51:20.647910 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:20.647864    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x" podStartSLOduration=1.647850544 podStartE2EDuration="1.647850544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:51:20.646915238 +0000 UTC m=+6909.210623648" watchObservedRunningTime="2026-11-02 13:51:20.647850544 +0000 UTC m=+6909.211558953"
Nov 02 13:51:21.537534 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:21.537501    4464 generic.go:356] "Generic (PLEG): container finished" podID="a14d2b91-407b-4843-af74-67b3298caf27" containerID="bd1916845e8e13b3112c21bc277fc8dc2dc3a9e50990891bc2a1c0769665abd4" exitCode=0
Nov 02 13:51:21.537916 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:21.537590    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x" event={"ID":"a14d2b91-407b-4843-af74-67b3298caf27","Type":"ContainerDied","Data":"bd1916845e8e13b3112c21bc277fc8dc2dc3a9e50990891bc2a1c0769665abd4"}
Nov 02 13:51:22.672670 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:22.672644    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"
Nov 02 13:51:22.775539 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:22.775504    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a14d2b91-407b-4843-af74-67b3298caf27-kube-api-access-mb2xl\" (UniqueName: \"kubernetes.io/projected/a14d2b91-407b-4843-af74-67b3298caf27-kube-api-access-mb2xl\") pod \"a14d2b91-407b-4843-af74-67b3298caf27\" (UID: \"a14d2b91-407b-4843-af74-67b3298caf27\") "
Nov 02 13:51:22.775720 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:22.775572    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a14d2b91-407b-4843-af74-67b3298caf27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a14d2b91-407b-4843-af74-67b3298caf27-maas-api-xks-ca\") pod \"a14d2b91-407b-4843-af74-67b3298caf27\" (UID: \"a14d2b91-407b-4843-af74-67b3298caf27\") "
Nov 02 13:51:22.775720 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:22.775601    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a14d2b91-407b-4843-af74-67b3298caf27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a14d2b91-407b-4843-af74-67b3298caf27-maas-api-openshift-ca\") pod \"a14d2b91-407b-4843-af74-67b3298caf27\" (UID: \"a14d2b91-407b-4843-af74-67b3298caf27\") "
Nov 02 13:51:22.776035 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:22.776011    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a14d2b91-407b-4843-af74-67b3298caf27-maas-api-openshift-ca" pod "a14d2b91-407b-4843-af74-67b3298caf27" (UID: "a14d2b91-407b-4843-af74-67b3298caf27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:51:22.777739 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:22.777715    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a14d2b91-407b-4843-af74-67b3298caf27-maas-api-xks-ca" pod "a14d2b91-407b-4843-af74-67b3298caf27" (UID: "a14d2b91-407b-4843-af74-67b3298caf27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:51:22.777802 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:22.777731    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14d2b91-407b-4843-af74-67b3298caf27-kube-api-access-mb2xl" pod "a14d2b91-407b-4843-af74-67b3298caf27" (UID: "a14d2b91-407b-4843-af74-67b3298caf27"). InnerVolumeSpecName "kube-api-access-mb2xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:51:22.876387 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:22.876312    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb2xl\" (UniqueName: \"kubernetes.io/projected/a14d2b91-407b-4843-af74-67b3298caf27-kube-api-access-mb2xl\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:51:22.876387 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:22.876341    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a14d2b91-407b-4843-af74-67b3298caf27-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:51:22.876387 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:22.876351    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a14d2b91-407b-4843-af74-67b3298caf27-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:51:23.548939 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:23.548909    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"
Nov 02 13:51:23.549122 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:23.548913    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x" event={"ID":"a14d2b91-407b-4843-af74-67b3298caf27","Type":"ContainerDied","Data":"8e91ab85f7782c14c98056893ba09b5fcbc3c60f1d123611c22748251e5dda94"}
Nov 02 13:51:23.549122 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:23.549056    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e91ab85f7782c14c98056893ba09b5fcbc3c60f1d123611c22748251e5dda94"
Nov 02 13:51:28.291154 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:51:28.291115    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:51:32.306120 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:32.306089    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:51:53.875249 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:53.875209    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"]
Nov 02 13:51:53.875842 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:53.875821    4464 cpu_manager.go:414] "containerMap: removing container" podUID="a14d2b91-407b-4843-af74-67b3298caf27" containerName="revoke-keys"
Nov 02 13:51:53.875938 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:53.875845    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a14d2b91-407b-4843-af74-67b3298caf27" containerName="revoke-keys"
Nov 02 13:51:53.876021 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:53.875962    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="a14d2b91-407b-4843-af74-67b3298caf27" containerName="revoke-keys"
Nov 02 13:51:53.879218 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:53.879197    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:53.926019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:53.925968    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"]
Nov 02 13:51:53.929515 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:53.929498    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:53.953309 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:53.953286    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d10b658-29b5-4e07-84df-13326972bead-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:53.953441 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:53.953325    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d10b658-29b5-4e07-84df-13326972bead-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:53.953441 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:53.953347    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d10b658-29b5-4e07-84df-13326972bead-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:53.953441 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:53.953365    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d10b658-29b5-4e07-84df-13326972bead-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:53.953566 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:53.953496    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d10b658-29b5-4e07-84df-13326972bead-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:53.953566 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:53.953535    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d10b658-29b5-4e07-84df-13326972bead-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:53.953566 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:53.953554    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d10b658-29b5-4e07-84df-13326972bead-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:53.953724 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:53.953589    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg8rx\" (UniqueName: \"kubernetes.io/projected/2d10b658-29b5-4e07-84df-13326972bead-kube-api-access-gg8rx\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:53.953724 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:53.953673    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d10b658-29b5-4e07-84df-13326972bead-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:53.981721 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:53.981693    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"]
Nov 02 13:51:54.048004 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.047953    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"]
Nov 02 13:51:54.054212 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054187    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91491db1-c3aa-422e-9e3d-5575b4197aec-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.054368 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054219    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91491db1-c3aa-422e-9e3d-5575b4197aec-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.054368 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054257    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91491db1-c3aa-422e-9e3d-5575b4197aec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.054486 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054371    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91491db1-c3aa-422e-9e3d-5575b4197aec-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.054486 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054396    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d10b658-29b5-4e07-84df-13326972bead-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.054486 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054429    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nvl8\" (UniqueName: \"kubernetes.io/projected/91491db1-c3aa-422e-9e3d-5575b4197aec-kube-api-access-7nvl8\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.054655 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054549    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91491db1-c3aa-422e-9e3d-5575b4197aec-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.054655 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054584    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d10b658-29b5-4e07-84df-13326972bead-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.054655 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054616    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d10b658-29b5-4e07-84df-13326972bead-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.054830 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054659    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91491db1-c3aa-422e-9e3d-5575b4197aec-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.054830 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054708    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d10b658-29b5-4e07-84df-13326972bead-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.054830 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054730    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91491db1-c3aa-422e-9e3d-5575b4197aec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.054830 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054764    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d10b658-29b5-4e07-84df-13326972bead-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.055102 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054855    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d10b658-29b5-4e07-84df-13326972bead-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.055102 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054872    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d10b658-29b5-4e07-84df-13326972bead-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.055102 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054919    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91491db1-c3aa-422e-9e3d-5575b4197aec-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.055102 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054950    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d10b658-29b5-4e07-84df-13326972bead-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.055102 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054962    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d10b658-29b5-4e07-84df-13326972bead-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.055102 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.054996    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg8rx\" (UniqueName: \"kubernetes.io/projected/2d10b658-29b5-4e07-84df-13326972bead-kube-api-access-gg8rx\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.055102 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.055034    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d10b658-29b5-4e07-84df-13326972bead-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.055480 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.055138    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d10b658-29b5-4e07-84df-13326972bead-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.055480 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.055404    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d10b658-29b5-4e07-84df-13326972bead-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.055647 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.055626    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d10b658-29b5-4e07-84df-13326972bead-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.056845 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.056826    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d10b658-29b5-4e07-84df-13326972bead-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.057753 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.057731    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d10b658-29b5-4e07-84df-13326972bead-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.116719 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.116683    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d10b658-29b5-4e07-84df-13326972bead-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.116860 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.116741    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg8rx\" (UniqueName: \"kubernetes.io/projected/2d10b658-29b5-4e07-84df-13326972bead-kube-api-access-gg8rx\") pod \"maas-default-gateway-openshift-default-6dcf757899-dghm7\" (UID: \"2d10b658-29b5-4e07-84df-13326972bead\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.156424 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.156355    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91491db1-c3aa-422e-9e3d-5575b4197aec-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.156424 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.156394    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91491db1-c3aa-422e-9e3d-5575b4197aec-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.156623 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.156441    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91491db1-c3aa-422e-9e3d-5575b4197aec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.156623 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.156483    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91491db1-c3aa-422e-9e3d-5575b4197aec-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.156623 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.156536    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nvl8\" (UniqueName: \"kubernetes.io/projected/91491db1-c3aa-422e-9e3d-5575b4197aec-kube-api-access-7nvl8\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.156623 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.156565    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91491db1-c3aa-422e-9e3d-5575b4197aec-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.156623 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.156605    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91491db1-c3aa-422e-9e3d-5575b4197aec-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.156910 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.156633    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91491db1-c3aa-422e-9e3d-5575b4197aec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.156910 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.156708    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91491db1-c3aa-422e-9e3d-5575b4197aec-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.156910 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.156753    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91491db1-c3aa-422e-9e3d-5575b4197aec-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.156910 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.156793    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91491db1-c3aa-422e-9e3d-5575b4197aec-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.156910 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.156870    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91491db1-c3aa-422e-9e3d-5575b4197aec-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.157242 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.157096    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91491db1-c3aa-422e-9e3d-5575b4197aec-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.157346 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.157329    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91491db1-c3aa-422e-9e3d-5575b4197aec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.158690 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.158670    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91491db1-c3aa-422e-9e3d-5575b4197aec-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.158792 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.158672    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91491db1-c3aa-422e-9e3d-5575b4197aec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.191714 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.191691    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:54.218211 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.218183    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91491db1-c3aa-422e-9e3d-5575b4197aec-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.218345 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.218308    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nvl8\" (UniqueName: \"kubernetes.io/projected/91491db1-c3aa-422e-9e3d-5575b4197aec-kube-api-access-7nvl8\") pod \"maas-default-gateway-openshift-default-6dcf757899-cnggl\" (UID: \"91491db1-c3aa-422e-9e3d-5575b4197aec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.242614 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.242585    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:54.615526 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.615501    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"]
Nov 02 13:51:54.616403 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:51:54.616377    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d10b658_29b5_4e07_84df_13326972bead.slice/crio-4a992b5a0a9a31f93e2100324ad0bcd25661bb7da45261c93108aff39ae345c0 WatchSource:0}: Error finding container 4a992b5a0a9a31f93e2100324ad0bcd25661bb7da45261c93108aff39ae345c0: Status 404 returned error can't find the container with id 4a992b5a0a9a31f93e2100324ad0bcd25661bb7da45261c93108aff39ae345c0
Nov 02 13:51:54.618404 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.618376    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:51:54.618489 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.618442    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:51:54.618489 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.618470    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:51:54.667222 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.667201    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"]
Nov 02 13:51:54.668595 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:51:54.668572    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91491db1_c3aa_422e_9e3d_5575b4197aec.slice/crio-5d71e74ed6b5ea437afafb6aebfcd278df3885a958a0543ccda3c5a6df603cc1 WatchSource:0}: Error finding container 5d71e74ed6b5ea437afafb6aebfcd278df3885a958a0543ccda3c5a6df603cc1: Status 404 returned error can't find the container with id 5d71e74ed6b5ea437afafb6aebfcd278df3885a958a0543ccda3c5a6df603cc1
Nov 02 13:51:54.670796 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.670769    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:51:54.670866 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.670838    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:51:54.670902 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.670877    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:51:54.700050 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.700018    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7" event={"ID":"2d10b658-29b5-4e07-84df-13326972bead","Type":"ContainerStarted","Data":"15ce588bc6d8bfa2d5336d376ef17fa5d73d68dee28866727b57becd3ca8f16b"}
Nov 02 13:51:54.700175 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.700083    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7" event={"ID":"2d10b658-29b5-4e07-84df-13326972bead","Type":"ContainerStarted","Data":"4a992b5a0a9a31f93e2100324ad0bcd25661bb7da45261c93108aff39ae345c0"}
Nov 02 13:51:54.701620 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.701593    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl" event={"ID":"91491db1-c3aa-422e-9e3d-5575b4197aec","Type":"ContainerStarted","Data":"5d71e74ed6b5ea437afafb6aebfcd278df3885a958a0543ccda3c5a6df603cc1"}
Nov 02 13:51:54.822265 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:54.822209    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7" podStartSLOduration=1.822191992 podStartE2EDuration="1.822191992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:51:54.822160731 +0000 UTC m=+6943.385869140" watchObservedRunningTime="2026-11-02 13:51:54.822191992 +0000 UTC m=+6943.385900401"
Nov 02 13:51:55.195998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:55.195353    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:51:55.204007 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:55.198381    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dghm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" start-of-body=
Nov 02 13:51:55.204007 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:55.198435    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7" podUID="2d10b658-29b5-4e07-84df-13326972bead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused"
Nov 02 13:51:55.714112 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:55.714067    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl" event={"ID":"91491db1-c3aa-422e-9e3d-5575b4197aec","Type":"ContainerStarted","Data":"b96e7dbd5331db81bf099ba81334dbcbeaaefcf55d7ea24c1108e4ea6d3834f1"}
Nov 02 13:51:55.851199 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:55.851146    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl" podStartSLOduration=2.851132744 podStartE2EDuration="2.851132744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:51:55.84946745 +0000 UTC m=+6944.413175851" watchObservedRunningTime="2026-11-02 13:51:55.851132744 +0000 UTC m=+6944.414841153"
Nov 02 13:51:56.196009 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:56.192407    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dghm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" start-of-body=
Nov 02 13:51:56.196009 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:56.192472    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7" podUID="2d10b658-29b5-4e07-84df-13326972bead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused"
Nov 02 13:51:56.242854 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:56.242826    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:51:56.244169 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:56.244145    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cnggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused" start-of-body=
Nov 02 13:51:56.244232 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:56.244191    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl" podUID="91491db1-c3aa-422e-9e3d-5575b4197aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused"
Nov 02 13:51:57.197000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:57.196082    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dghm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" start-of-body=
Nov 02 13:51:57.197000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:57.196156    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7" podUID="2d10b658-29b5-4e07-84df-13326972bead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused"
Nov 02 13:51:57.243605 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:57.243563    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cnggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused" start-of-body=
Nov 02 13:51:57.243761 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:57.243634    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl" podUID="91491db1-c3aa-422e-9e3d-5575b4197aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused"
Nov 02 13:51:58.195999 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:58.195436    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dghm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" start-of-body=
Nov 02 13:51:58.195999 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:58.195499    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7" podUID="2d10b658-29b5-4e07-84df-13326972bead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused"
Nov 02 13:51:58.243560 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:58.243526    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cnggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused" start-of-body=
Nov 02 13:51:58.243927 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:58.243580    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl" podUID="91491db1-c3aa-422e-9e3d-5575b4197aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused"
Nov 02 13:51:58.333034 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:51:58.331522    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:51:59.192382 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:59.192343    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dghm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" start-of-body=
Nov 02 13:51:59.192682 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:59.192655    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7" podUID="2d10b658-29b5-4e07-84df-13326972bead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused"
Nov 02 13:51:59.242910 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:59.242882    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cnggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused" start-of-body=
Nov 02 13:51:59.243200 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:51:59.243063    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl" podUID="91491db1-c3aa-422e-9e3d-5575b4197aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused"
Nov 02 13:52:00.195414 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:00.195374    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dghm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" start-of-body=
Nov 02 13:52:00.196057 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:00.196025    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7" podUID="2d10b658-29b5-4e07-84df-13326972bead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused"
Nov 02 13:52:00.243797 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:00.243760    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cnggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused" start-of-body=
Nov 02 13:52:00.243955 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:00.243814    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl" podUID="91491db1-c3aa-422e-9e3d-5575b4197aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused"
Nov 02 13:52:01.193018 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:01.192964    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dghm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" start-of-body=
Nov 02 13:52:01.193204 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:01.193047    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7" podUID="2d10b658-29b5-4e07-84df-13326972bead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused"
Nov 02 13:52:01.243148 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:01.243112    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cnggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused" start-of-body=
Nov 02 13:52:01.243525 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:01.243173    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl" podUID="91491db1-c3aa-422e-9e3d-5575b4197aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused"
Nov 02 13:52:02.192493 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:02.192451    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dghm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused" start-of-body=
Nov 02 13:52:02.200562 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:02.192951    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7" podUID="2d10b658-29b5-4e07-84df-13326972bead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": dial tcp 10.133.1.80:15021: connect: connection refused"
Nov 02 13:52:02.243735 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:02.243698    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cnggl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused" start-of-body=
Nov 02 13:52:02.244113 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:02.243761    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl" podUID="91491db1-c3aa-422e-9e3d-5575b4197aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.81:15021/healthz/ready\": dial tcp 10.133.1.81:15021: connect: connection refused"
Nov 02 13:52:03.838364 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:03.838332    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:52:03.839390 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:03.839362    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:52:03.839957 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:03.839937    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7"
Nov 02 13:52:03.994680 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:03.994647    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:52:03.995008 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:03.994963    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:52:03.995788 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:52:03.995768    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl"
Nov 02 13:52:28.290486 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:52:28.290452    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:52:58.291077 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:52:58.291040    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:53:01.410161 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.410129    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"]
Nov 02 13:53:01.413727 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.413706    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.468220 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.468189    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8869c0de-openshift-default-dockercfg-7p8kx\""
Nov 02 13:53:01.527475 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.527432    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istiod-ca-cert\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.527475 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.527486    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-workload-certs\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.527700 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.527558    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-credential-socket\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.527700 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.527598    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-data\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.527700 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.527619    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-token\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.527700 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.527676    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-workload-socket\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.527860 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.527708    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-podinfo\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.527860 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.527726    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlgsq\" (UniqueName: \"kubernetes.io/projected/15b34b5d-79c7-4d32-b72d-9d6c398257a6-kube-api-access-zlgsq\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.527860 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.527800    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-envoy\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.537067 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.537033    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"]
Nov 02 13:53:01.628675 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.628642    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-workload-socket\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.628675 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.628680    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-podinfo\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.628898 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.628709    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlgsq\" (UniqueName: \"kubernetes.io/projected/15b34b5d-79c7-4d32-b72d-9d6c398257a6-kube-api-access-zlgsq\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.628898 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.628755    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-envoy\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.628898 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.628859    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istiod-ca-cert\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.629087 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.628900    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-workload-certs\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.629087 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.628946    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-credential-socket\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.629087 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.629013    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-data\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.629087 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.629038    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-token\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.629340 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.629148    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-workload-socket\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.629458 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.629430    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-credential-socket\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.629542 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.629441    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-data\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.629542 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.629530    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-workload-certs\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.629792 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.629770    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istiod-ca-cert\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.631120 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.631100    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-envoy\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.631335 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.631318    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-podinfo\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.691340 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.691281    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlgsq\" (UniqueName: \"kubernetes.io/projected/15b34b5d-79c7-4d32-b72d-9d6c398257a6-kube-api-access-zlgsq\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.691340 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.691285    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-token\") pod \"e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") " pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:01.726971 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:01.726948    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:02.165610 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:02.165578    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"]
Nov 02 13:53:02.166771 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:53:02.166742    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b34b5d_79c7_4d32_b72d_9d6c398257a6.slice/crio-ef899b4f3276f943f8446991adcaf92392c91025e96d9d1eae5f5c4affc2ef64 WatchSource:0}: Error finding container ef899b4f3276f943f8446991adcaf92392c91025e96d9d1eae5f5c4affc2ef64: Status 404 returned error can't find the container with id ef899b4f3276f943f8446991adcaf92392c91025e96d9d1eae5f5c4affc2ef64
Nov 02 13:53:02.168863 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:02.168833    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:02.168968 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:02.168906    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:02.168968 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:02.168948    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:03.047604 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:03.047561    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv" event={"ID":"15b34b5d-79c7-4d32-b72d-9d6c398257a6","Type":"ContainerStarted","Data":"957ff6f458bf813795c5bb7f14d81c9db73cefdec33170209fcaaa3ff417e097"}
Nov 02 13:53:03.048034 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:03.047645    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv" event={"ID":"15b34b5d-79c7-4d32-b72d-9d6c398257a6","Type":"ContainerStarted","Data":"ef899b4f3276f943f8446991adcaf92392c91025e96d9d1eae5f5c4affc2ef64"}
Nov 02 13:53:03.221223 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:03.221170    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv" podStartSLOduration=2.221155691 podStartE2EDuration="2.221155691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:03.219863021 +0000 UTC m=+7011.783571432" watchObservedRunningTime="2026-11-02 13:53:03.221155691 +0000 UTC m=+7011.784864077"
Nov 02 13:53:03.727454 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:03.727424    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:03.732293 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:03.732270    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:04.052810 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:04.052722    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:04.053552 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:04.053533    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:09.864606 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:09.864567    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"]
Nov 02 13:53:09.868455 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:09.868431    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"
Nov 02 13:53:09.903763 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:09.903733    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985cb7db-44a3-4a5d-88d0-e02d0b98bc3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8\" (UID: \"985cb7db-44a3-4a5d-88d0-e02d0b98bc3e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"
Nov 02 13:53:09.903919 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:09.903881    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4ggt\" (UniqueName: \"kubernetes.io/projected/985cb7db-44a3-4a5d-88d0-e02d0b98bc3e-kube-api-access-b4ggt\") pod \"payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8\" (UID: \"985cb7db-44a3-4a5d-88d0-e02d0b98bc3e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"
Nov 02 13:53:09.929604 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:09.929575    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"]
Nov 02 13:53:09.981343 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:09.981311    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8869c0de-dockercfg-9gzfl\""
Nov 02 13:53:10.007258 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:09.995196    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8869c0de\""
Nov 02 13:53:10.010939 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:10.010875    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985cb7db-44a3-4a5d-88d0-e02d0b98bc3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8\" (UID: \"985cb7db-44a3-4a5d-88d0-e02d0b98bc3e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"
Nov 02 13:53:10.011320 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:10.011280    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4ggt\" (UniqueName: \"kubernetes.io/projected/985cb7db-44a3-4a5d-88d0-e02d0b98bc3e-kube-api-access-b4ggt\") pod \"payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8\" (UID: \"985cb7db-44a3-4a5d-88d0-e02d0b98bc3e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"
Nov 02 13:53:10.012555 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:10.012528    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985cb7db-44a3-4a5d-88d0-e02d0b98bc3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8\" (UID: \"985cb7db-44a3-4a5d-88d0-e02d0b98bc3e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"
Nov 02 13:53:10.034206 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:10.034169    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"]
Nov 02 13:53:10.037894 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:10.037872    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"
Nov 02 13:53:10.074941 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:10.074911    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"]
Nov 02 13:53:10.075654 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:10.075629    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4ggt\" (UniqueName: \"kubernetes.io/projected/985cb7db-44a3-4a5d-88d0-e02d0b98bc3e-kube-api-access-b4ggt\") pod \"payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8\" (UID: \"985cb7db-44a3-4a5d-88d0-e02d0b98bc3e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"
Nov 02 13:53:10.112446 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:10.112410    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvln5\" (UniqueName: \"kubernetes.io/projected/52d60cf1-1f4e-4436-9f0f-270fed53b0b4-kube-api-access-lvln5\") pod \"payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6\" (UID: \"52d60cf1-1f4e-4436-9f0f-270fed53b0b4\") " pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"
Nov 02 13:53:10.112618 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:10.112549    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d60cf1-1f4e-4436-9f0f-270fed53b0b4-plugins-config-volume\") pod \"payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6\" (UID: \"52d60cf1-1f4e-4436-9f0f-270fed53b0b4\") " pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"
Nov 02 13:53:10.182091 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:10.182018    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"
Nov 02 13:53:10.213936 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:10.213897    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvln5\" (UniqueName: \"kubernetes.io/projected/52d60cf1-1f4e-4436-9f0f-270fed53b0b4-kube-api-access-lvln5\") pod \"payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6\" (UID: \"52d60cf1-1f4e-4436-9f0f-270fed53b0b4\") " pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"
Nov 02 13:53:10.214079 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:10.213985    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d60cf1-1f4e-4436-9f0f-270fed53b0b4-plugins-config-volume\") pod \"payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6\" (UID: \"52d60cf1-1f4e-4436-9f0f-270fed53b0b4\") " pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"
Nov 02 13:53:10.214658 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:10.214622    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d60cf1-1f4e-4436-9f0f-270fed53b0b4-plugins-config-volume\") pod \"payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6\" (UID: \"52d60cf1-1f4e-4436-9f0f-270fed53b0b4\") " pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"
Nov 02 13:53:10.275656 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:10.275628    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvln5\" (UniqueName: \"kubernetes.io/projected/52d60cf1-1f4e-4436-9f0f-270fed53b0b4-kube-api-access-lvln5\") pod \"payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6\" (UID: \"52d60cf1-1f4e-4436-9f0f-270fed53b0b4\") " pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"
Nov 02 13:53:10.350806 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:10.350780    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"
Nov 02 13:53:10.557160 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:10.557129    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"]
Nov 02 13:53:10.558184 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:53:10.558161    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985cb7db_44a3_4a5d_88d0_e02d0b98bc3e.slice/crio-0b7afc06bdc8fa6546f8047c6f3e3e659181e86b23a4ce50ab00a87b1dff0600 WatchSource:0}: Error finding container 0b7afc06bdc8fa6546f8047c6f3e3e659181e86b23a4ce50ab00a87b1dff0600: Status 404 returned error can't find the container with id 0b7afc06bdc8fa6546f8047c6f3e3e659181e86b23a4ce50ab00a87b1dff0600
Nov 02 13:53:10.724627 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:10.724601    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"]
Nov 02 13:53:10.727022 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:53:10.726994    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d60cf1_1f4e_4436_9f0f_270fed53b0b4.slice/crio-0e73cd6b5e6b96b21ce932b8fc86cc5916c136ce298ef0165cbdb9c0faa79b33 WatchSource:0}: Error finding container 0e73cd6b5e6b96b21ce932b8fc86cc5916c136ce298ef0165cbdb9c0faa79b33: Status 404 returned error can't find the container with id 0e73cd6b5e6b96b21ce932b8fc86cc5916c136ce298ef0165cbdb9c0faa79b33
Nov 02 13:53:11.090616 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:11.090579    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6" event={"ID":"52d60cf1-1f4e-4436-9f0f-270fed53b0b4","Type":"ContainerStarted","Data":"0843f2f2b14170d0d83cb500216bb6970c2c76d1cc53ec3574ad8cca4ff52627"}
Nov 02 13:53:11.091114 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:11.090645    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6" event={"ID":"52d60cf1-1f4e-4436-9f0f-270fed53b0b4","Type":"ContainerStarted","Data":"0e73cd6b5e6b96b21ce932b8fc86cc5916c136ce298ef0165cbdb9c0faa79b33"}
Nov 02 13:53:11.091114 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:11.090671    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"
Nov 02 13:53:11.092106 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:11.092081    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8" event={"ID":"985cb7db-44a3-4a5d-88d0-e02d0b98bc3e","Type":"ContainerStarted","Data":"8fbfe2a405b3654579f1a9363cdf4614109c3ef6f218be0bcf1370a81e6c876d"}
Nov 02 13:53:11.092230 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:11.092135    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"
Nov 02 13:53:11.092230 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:11.092158    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8" event={"ID":"985cb7db-44a3-4a5d-88d0-e02d0b98bc3e","Type":"ContainerStarted","Data":"0b7afc06bdc8fa6546f8047c6f3e3e659181e86b23a4ce50ab00a87b1dff0600"}
Nov 02 13:53:11.210303 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:11.210248    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6" podStartSLOduration=2.210234094 podStartE2EDuration="2.210234094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:11.208588331 +0000 UTC m=+7019.772296738" watchObservedRunningTime="2026-11-02 13:53:11.210234094 +0000 UTC m=+7019.773942521"
Nov 02 13:53:20.622023 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:20.617921    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8" podStartSLOduration=11.617899738 podStartE2EDuration="11.617899738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:11.332698895 +0000 UTC m=+7019.896407305" watchObservedRunningTime="2026-11-02 13:53:20.617899738 +0000 UTC m=+7029.181608151"
Nov 02 13:53:20.622877 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:20.622843    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:53:22.100323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:22.100295    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"
Nov 02 13:53:22.100805 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:22.100468    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"
Nov 02 13:53:28.295158 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:53:28.295112    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:53:30.208348 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:30.208264    4464 generic.go:356] "Generic (PLEG): container finished" podID="15b34b5d-79c7-4d32-b72d-9d6c398257a6" containerID="957ff6f458bf813795c5bb7f14d81c9db73cefdec33170209fcaaa3ff417e097" exitCode=0
Nov 02 13:53:30.208766 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:30.208339    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv" event={"ID":"15b34b5d-79c7-4d32-b72d-9d6c398257a6","Type":"ContainerDied","Data":"957ff6f458bf813795c5bb7f14d81c9db73cefdec33170209fcaaa3ff417e097"}
Nov 02 13:53:30.208766 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:30.208724    4464 scope.go:122] "RemoveContainer" containerID="957ff6f458bf813795c5bb7f14d81c9db73cefdec33170209fcaaa3ff417e097"
Nov 02 13:53:30.209918 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:30.209893    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:30.209997 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:30.209961    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:30.210044 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:30.210018    4464 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:31.215769 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:31.215737    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv" event={"ID":"15b34b5d-79c7-4d32-b72d-9d6c398257a6","Type":"ContainerStarted","Data":"9379cc8494601be75afe7f2cfba085478d72d0c8d87efea82ed4527869131166"}
Nov 02 13:53:31.727300 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:31.727264    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:31.728397 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:31.728374    4464 patch_prober.go:29] interesting pod/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused" start-of-body=
Nov 02 13:53:31.728492 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:31.728432    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv" podUID="15b34b5d-79c7-4d32-b72d-9d6c398257a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused"
Nov 02 13:53:32.728096 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:32.728061    4464 patch_prober.go:29] interesting pod/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused" start-of-body=
Nov 02 13:53:32.728495 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:32.728112    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv" podUID="15b34b5d-79c7-4d32-b72d-9d6c398257a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused"
Nov 02 13:53:33.728289 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:33.728248    4464 patch_prober.go:29] interesting pod/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused" start-of-body=
Nov 02 13:53:33.728781 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:33.728310    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv" podUID="15b34b5d-79c7-4d32-b72d-9d6c398257a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused"
Nov 02 13:53:34.052919 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:34.052839    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:34.727416 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:34.727381    4464 patch_prober.go:29] interesting pod/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused" start-of-body=
Nov 02 13:53:34.727601 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:34.727434    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv" podUID="15b34b5d-79c7-4d32-b72d-9d6c398257a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused"
Nov 02 13:53:35.727944 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:35.727907    4464 patch_prober.go:29] interesting pod/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused" start-of-body=
Nov 02 13:53:35.728380 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:35.727961    4464 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv" podUID="15b34b5d-79c7-4d32-b72d-9d6c398257a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused"
Nov 02 13:53:36.732355 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:36.732323    4464 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:36.733388 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:53:36.733366    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:53:58.291269 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:53:58.291192    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:54:28.291182 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:54:28.291134    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:54:42.061297 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:54:42.060821    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"]
Nov 02 13:54:42.109956 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:54:42.109925    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5206625f-cpj4j"]
Nov 02 13:54:43.580844 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:54:43.580811    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf62f22d-f314-4c99-a0f7-6ffd807c5148" path="/var/lib/kubelet/pods/cf62f22d-f314-4c99-a0f7-6ffd807c5148/volumes"
Nov 02 13:54:58.290961 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:54:58.290927    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:55:08.983223 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:08.983192    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"]
Nov 02 13:55:09.093406 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:09.093375    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"]
Nov 02 13:55:13.983702 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:13.983641    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6" podUID="52d60cf1-1f4e-4436-9f0f-270fed53b0b4" containerName="payload-processing" containerID="cri-o://0843f2f2b14170d0d83cb500216bb6970c2c76d1cc53ec3574ad8cca4ff52627" gracePeriod=25
Nov 02 13:55:14.094184 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.094144    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8" podUID="985cb7db-44a3-4a5d-88d0-e02d0b98bc3e" containerName="payload-pre-processing" containerID="cri-o://8fbfe2a405b3654579f1a9363cdf4614109c3ef6f218be0bcf1370a81e6c876d" gracePeriod=25
Nov 02 13:55:14.233546 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.233520    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"
Nov 02 13:55:14.342785 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.342762    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"
Nov 02 13:55:14.373499 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.373461    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d60cf1-1f4e-4436-9f0f-270fed53b0b4-kube-api-access-lvln5\" (UniqueName: \"kubernetes.io/projected/52d60cf1-1f4e-4436-9f0f-270fed53b0b4-kube-api-access-lvln5\") pod \"52d60cf1-1f4e-4436-9f0f-270fed53b0b4\" (UID: \"52d60cf1-1f4e-4436-9f0f-270fed53b0b4\") "
Nov 02 13:55:14.373499 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.373497    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52d60cf1-1f4e-4436-9f0f-270fed53b0b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d60cf1-1f4e-4436-9f0f-270fed53b0b4-plugins-config-volume\") pod \"52d60cf1-1f4e-4436-9f0f-270fed53b0b4\" (UID: \"52d60cf1-1f4e-4436-9f0f-270fed53b0b4\") "
Nov 02 13:55:14.373997 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.373952    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d60cf1-1f4e-4436-9f0f-270fed53b0b4-plugins-config-volume" pod "52d60cf1-1f4e-4436-9f0f-270fed53b0b4" (UID: "52d60cf1-1f4e-4436-9f0f-270fed53b0b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:55:14.375545 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.375525    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d60cf1-1f4e-4436-9f0f-270fed53b0b4-kube-api-access-lvln5" pod "52d60cf1-1f4e-4436-9f0f-270fed53b0b4" (UID: "52d60cf1-1f4e-4436-9f0f-270fed53b0b4"). InnerVolumeSpecName "kube-api-access-lvln5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:55:14.480586 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.480544    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/985cb7db-44a3-4a5d-88d0-e02d0b98bc3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985cb7db-44a3-4a5d-88d0-e02d0b98bc3e-plugins-config-volume\") pod \"985cb7db-44a3-4a5d-88d0-e02d0b98bc3e\" (UID: \"985cb7db-44a3-4a5d-88d0-e02d0b98bc3e\") "
Nov 02 13:55:14.480785 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.480623    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/985cb7db-44a3-4a5d-88d0-e02d0b98bc3e-kube-api-access-b4ggt\" (UniqueName: \"kubernetes.io/projected/985cb7db-44a3-4a5d-88d0-e02d0b98bc3e-kube-api-access-b4ggt\") pod \"985cb7db-44a3-4a5d-88d0-e02d0b98bc3e\" (UID: \"985cb7db-44a3-4a5d-88d0-e02d0b98bc3e\") "
Nov 02 13:55:14.481099 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.481078    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvln5\" (UniqueName: \"kubernetes.io/projected/52d60cf1-1f4e-4436-9f0f-270fed53b0b4-kube-api-access-lvln5\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:55:14.481212 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.481102    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52d60cf1-1f4e-4436-9f0f-270fed53b0b4-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:55:14.481779 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.481751    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985cb7db-44a3-4a5d-88d0-e02d0b98bc3e-plugins-config-volume" pod "985cb7db-44a3-4a5d-88d0-e02d0b98bc3e" (UID: "985cb7db-44a3-4a5d-88d0-e02d0b98bc3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:55:14.483861 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.483838    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985cb7db-44a3-4a5d-88d0-e02d0b98bc3e-kube-api-access-b4ggt" pod "985cb7db-44a3-4a5d-88d0-e02d0b98bc3e" (UID: "985cb7db-44a3-4a5d-88d0-e02d0b98bc3e"). InnerVolumeSpecName "kube-api-access-b4ggt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:55:14.582427 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.582349    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4ggt\" (UniqueName: \"kubernetes.io/projected/985cb7db-44a3-4a5d-88d0-e02d0b98bc3e-kube-api-access-b4ggt\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:55:14.582427 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.582376    4464 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985cb7db-44a3-4a5d-88d0-e02d0b98bc3e-plugins-config-volume\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:55:14.730217 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.730185    4464 generic.go:356] "Generic (PLEG): container finished" podID="52d60cf1-1f4e-4436-9f0f-270fed53b0b4" containerID="0843f2f2b14170d0d83cb500216bb6970c2c76d1cc53ec3574ad8cca4ff52627" exitCode=0
Nov 02 13:55:14.730420 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.730243    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"
Nov 02 13:55:14.730420 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.730275    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6" event={"ID":"52d60cf1-1f4e-4436-9f0f-270fed53b0b4","Type":"ContainerDied","Data":"0843f2f2b14170d0d83cb500216bb6970c2c76d1cc53ec3574ad8cca4ff52627"}
Nov 02 13:55:14.730420 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.730339    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6" event={"ID":"52d60cf1-1f4e-4436-9f0f-270fed53b0b4","Type":"ContainerDied","Data":"0e73cd6b5e6b96b21ce932b8fc86cc5916c136ce298ef0165cbdb9c0faa79b33"}
Nov 02 13:55:14.730420 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.730370    4464 scope.go:122] "RemoveContainer" containerID="0843f2f2b14170d0d83cb500216bb6970c2c76d1cc53ec3574ad8cca4ff52627"
Nov 02 13:55:14.731871 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.731844    4464 generic.go:356] "Generic (PLEG): container finished" podID="985cb7db-44a3-4a5d-88d0-e02d0b98bc3e" containerID="8fbfe2a405b3654579f1a9363cdf4614109c3ef6f218be0bcf1370a81e6c876d" exitCode=0
Nov 02 13:55:14.731998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.731876    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8" event={"ID":"985cb7db-44a3-4a5d-88d0-e02d0b98bc3e","Type":"ContainerDied","Data":"8fbfe2a405b3654579f1a9363cdf4614109c3ef6f218be0bcf1370a81e6c876d"}
Nov 02 13:55:14.731998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.731908    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"
Nov 02 13:55:14.731998 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.731935    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8" event={"ID":"985cb7db-44a3-4a5d-88d0-e02d0b98bc3e","Type":"ContainerDied","Data":"0b7afc06bdc8fa6546f8047c6f3e3e659181e86b23a4ce50ab00a87b1dff0600"}
Nov 02 13:55:14.743122 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.743106    4464 scope.go:122] "RemoveContainer" containerID="0843f2f2b14170d0d83cb500216bb6970c2c76d1cc53ec3574ad8cca4ff52627"
Nov 02 13:55:14.743465 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:55:14.743437    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0843f2f2b14170d0d83cb500216bb6970c2c76d1cc53ec3574ad8cca4ff52627\": container with ID starting with 0843f2f2b14170d0d83cb500216bb6970c2c76d1cc53ec3574ad8cca4ff52627 not found: ID does not exist" containerID="0843f2f2b14170d0d83cb500216bb6970c2c76d1cc53ec3574ad8cca4ff52627"
Nov 02 13:55:14.743578 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.743471    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0843f2f2b14170d0d83cb500216bb6970c2c76d1cc53ec3574ad8cca4ff52627"} err="failed to get container status \"0843f2f2b14170d0d83cb500216bb6970c2c76d1cc53ec3574ad8cca4ff52627\": rpc error: code = NotFound desc = could not find container \"0843f2f2b14170d0d83cb500216bb6970c2c76d1cc53ec3574ad8cca4ff52627\": container with ID starting with 0843f2f2b14170d0d83cb500216bb6970c2c76d1cc53ec3574ad8cca4ff52627 not found: ID does not exist"
Nov 02 13:55:14.743578 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.743494    4464 scope.go:122] "RemoveContainer" containerID="8fbfe2a405b3654579f1a9363cdf4614109c3ef6f218be0bcf1370a81e6c876d"
Nov 02 13:55:14.752135 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.752109    4464 scope.go:122] "RemoveContainer" containerID="8fbfe2a405b3654579f1a9363cdf4614109c3ef6f218be0bcf1370a81e6c876d"
Nov 02 13:55:14.752384 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:55:14.752365    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fbfe2a405b3654579f1a9363cdf4614109c3ef6f218be0bcf1370a81e6c876d\": container with ID starting with 8fbfe2a405b3654579f1a9363cdf4614109c3ef6f218be0bcf1370a81e6c876d not found: ID does not exist" containerID="8fbfe2a405b3654579f1a9363cdf4614109c3ef6f218be0bcf1370a81e6c876d"
Nov 02 13:55:14.752435 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.752393    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fbfe2a405b3654579f1a9363cdf4614109c3ef6f218be0bcf1370a81e6c876d"} err="failed to get container status \"8fbfe2a405b3654579f1a9363cdf4614109c3ef6f218be0bcf1370a81e6c876d\": rpc error: code = NotFound desc = could not find container \"8fbfe2a405b3654579f1a9363cdf4614109c3ef6f218be0bcf1370a81e6c876d\": container with ID starting with 8fbfe2a405b3654579f1a9363cdf4614109c3ef6f218be0bcf1370a81e6c876d not found: ID does not exist"
Nov 02 13:55:14.897491 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.897396    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"]
Nov 02 13:55:14.901702 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:14.901678    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8869c0de-7679b957dc-m6nv8"]
Nov 02 13:55:15.014270 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:15.014235    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"]
Nov 02 13:55:15.019283 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:15.019259    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8869c0de-bf7df7f85-zwjd6"]
Nov 02 13:55:15.495002 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:15.486284    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:55:15.581291 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:15.581262    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52d60cf1-1f4e-4436-9f0f-270fed53b0b4" path="/var/lib/kubelet/pods/52d60cf1-1f4e-4436-9f0f-270fed53b0b4/volumes"
Nov 02 13:55:15.581638 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:15.581626    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="985cb7db-44a3-4a5d-88d0-e02d0b98bc3e" path="/var/lib/kubelet/pods/985cb7db-44a3-4a5d-88d0-e02d0b98bc3e/volumes"
Nov 02 13:55:24.312287 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:24.312256    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"]
Nov 02 13:55:24.312789 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:24.312454    4464 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv" podUID="15b34b5d-79c7-4d32-b72d-9d6c398257a6" containerName="istio-proxy" containerID="cri-o://9379cc8494601be75afe7f2cfba085478d72d0c8d87efea82ed4527869131166" gracePeriod=30
Nov 02 13:55:28.291232 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:55:28.291196    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:55:29.567846 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.567819    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:55:29.734491 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.734462    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-workload-socket\") pod \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") "
Nov 02 13:55:29.734674 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.734497    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-workload-certs\") pod \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") "
Nov 02 13:55:29.734674 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.734546    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b34b5d-79c7-4d32-b72d-9d6c398257a6-kube-api-access-zlgsq\" (UniqueName: \"kubernetes.io/projected/15b34b5d-79c7-4d32-b72d-9d6c398257a6-kube-api-access-zlgsq\") pod \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") "
Nov 02 13:55:29.734674 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.734604    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-envoy\") pod \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") "
Nov 02 13:55:29.734674 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.734650    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-podinfo\") pod \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") "
Nov 02 13:55:29.734876 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.734686    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-data\") pod \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") "
Nov 02 13:55:29.734876 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.734755    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-token\" (UniqueName: \"kubernetes.io/projected/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-token\") pod \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") "
Nov 02 13:55:29.734876 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.734753    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-workload-socket" pod "15b34b5d-79c7-4d32-b72d-9d6c398257a6" (UID: "15b34b5d-79c7-4d32-b72d-9d6c398257a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:55:29.734876 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.734857    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istiod-ca-cert\") pod \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") "
Nov 02 13:55:29.735118 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.734884    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-workload-certs" pod "15b34b5d-79c7-4d32-b72d-9d6c398257a6" (UID: "15b34b5d-79c7-4d32-b72d-9d6c398257a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:55:29.735118 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.734907    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-credential-socket\") pod \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\" (UID: \"15b34b5d-79c7-4d32-b72d-9d6c398257a6\") "
Nov 02 13:55:29.735234 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.735146    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-data" pod "15b34b5d-79c7-4d32-b72d-9d6c398257a6" (UID: "15b34b5d-79c7-4d32-b72d-9d6c398257a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:55:29.735295 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.735239    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istiod-ca-cert" pod "15b34b5d-79c7-4d32-b72d-9d6c398257a6" (UID: "15b34b5d-79c7-4d32-b72d-9d6c398257a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:55:29.735395 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.735370    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-credential-socket" pod "15b34b5d-79c7-4d32-b72d-9d6c398257a6" (UID: "15b34b5d-79c7-4d32-b72d-9d6c398257a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:55:29.735810 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.735643    4464 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-data\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:55:29.735810 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.735667    4464 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istiod-ca-cert\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:55:29.735810 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.735685    4464 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-credential-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:55:29.735810 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.735699    4464 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-workload-socket\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:55:29.735810 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.735714    4464 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-workload-certs\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:55:29.737019 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.736994    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-token" pod "15b34b5d-79c7-4d32-b72d-9d6c398257a6" (UID: "15b34b5d-79c7-4d32-b72d-9d6c398257a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:55:29.737236 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.737220    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-envoy" pod "15b34b5d-79c7-4d32-b72d-9d6c398257a6" (UID: "15b34b5d-79c7-4d32-b72d-9d6c398257a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:55:29.737350 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.737330    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b34b5d-79c7-4d32-b72d-9d6c398257a6-kube-api-access-zlgsq" pod "15b34b5d-79c7-4d32-b72d-9d6c398257a6" (UID: "15b34b5d-79c7-4d32-b72d-9d6c398257a6"). InnerVolumeSpecName "kube-api-access-zlgsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:55:29.737409 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.737330    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-podinfo" pod "15b34b5d-79c7-4d32-b72d-9d6c398257a6" (UID: "15b34b5d-79c7-4d32-b72d-9d6c398257a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:55:29.805426 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.805393    4464 generic.go:356] "Generic (PLEG): container finished" podID="15b34b5d-79c7-4d32-b72d-9d6c398257a6" containerID="9379cc8494601be75afe7f2cfba085478d72d0c8d87efea82ed4527869131166" exitCode=0
Nov 02 13:55:29.805582 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.805444    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"
Nov 02 13:55:29.805582 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.805470    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv" event={"ID":"15b34b5d-79c7-4d32-b72d-9d6c398257a6","Type":"ContainerDied","Data":"9379cc8494601be75afe7f2cfba085478d72d0c8d87efea82ed4527869131166"}
Nov 02 13:55:29.805582 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.805541    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv" event={"ID":"15b34b5d-79c7-4d32-b72d-9d6c398257a6","Type":"ContainerDied","Data":"ef899b4f3276f943f8446991adcaf92392c91025e96d9d1eae5f5c4affc2ef64"}
Nov 02 13:55:29.805698 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.805567    4464 scope.go:122] "RemoveContainer" containerID="9379cc8494601be75afe7f2cfba085478d72d0c8d87efea82ed4527869131166"
Nov 02 13:55:29.815047 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.815030    4464 scope.go:122] "RemoveContainer" containerID="957ff6f458bf813795c5bb7f14d81c9db73cefdec33170209fcaaa3ff417e097"
Nov 02 13:55:29.823088 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.823069    4464 scope.go:122] "RemoveContainer" containerID="9379cc8494601be75afe7f2cfba085478d72d0c8d87efea82ed4527869131166"
Nov 02 13:55:29.823357 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:55:29.823334    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9379cc8494601be75afe7f2cfba085478d72d0c8d87efea82ed4527869131166\": container with ID starting with 9379cc8494601be75afe7f2cfba085478d72d0c8d87efea82ed4527869131166 not found: ID does not exist" containerID="9379cc8494601be75afe7f2cfba085478d72d0c8d87efea82ed4527869131166"
Nov 02 13:55:29.823466 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.823365    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9379cc8494601be75afe7f2cfba085478d72d0c8d87efea82ed4527869131166"} err="failed to get container status \"9379cc8494601be75afe7f2cfba085478d72d0c8d87efea82ed4527869131166\": rpc error: code = NotFound desc = could not find container \"9379cc8494601be75afe7f2cfba085478d72d0c8d87efea82ed4527869131166\": container with ID starting with 9379cc8494601be75afe7f2cfba085478d72d0c8d87efea82ed4527869131166 not found: ID does not exist"
Nov 02 13:55:29.823466 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.823381    4464 scope.go:122] "RemoveContainer" containerID="957ff6f458bf813795c5bb7f14d81c9db73cefdec33170209fcaaa3ff417e097"
Nov 02 13:55:29.823589 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:55:29.823575    4464 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"957ff6f458bf813795c5bb7f14d81c9db73cefdec33170209fcaaa3ff417e097\": container with ID starting with 957ff6f458bf813795c5bb7f14d81c9db73cefdec33170209fcaaa3ff417e097 not found: ID does not exist" containerID="957ff6f458bf813795c5bb7f14d81c9db73cefdec33170209fcaaa3ff417e097"
Nov 02 13:55:29.823631 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.823595    4464 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"957ff6f458bf813795c5bb7f14d81c9db73cefdec33170209fcaaa3ff417e097"} err="failed to get container status \"957ff6f458bf813795c5bb7f14d81c9db73cefdec33170209fcaaa3ff417e097\": rpc error: code = NotFound desc = could not find container \"957ff6f458bf813795c5bb7f14d81c9db73cefdec33170209fcaaa3ff417e097\": container with ID starting with 957ff6f458bf813795c5bb7f14d81c9db73cefdec33170209fcaaa3ff417e097 not found: ID does not exist"
Nov 02 13:55:29.836738 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.836718    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlgsq\" (UniqueName: \"kubernetes.io/projected/15b34b5d-79c7-4d32-b72d-9d6c398257a6-kube-api-access-zlgsq\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:55:29.836825 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.836740    4464 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-envoy\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:55:29.836825 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.836750    4464 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-podinfo\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:55:29.836825 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.836759    4464 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15b34b5d-79c7-4d32-b72d-9d6c398257a6-istio-token\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:55:29.972434 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.972398    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"]
Nov 02 13:55:29.979218 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:29.979194    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8869c0de-openshift-default-56fcb59f45-8qgnv"]
Nov 02 13:55:31.581948 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:31.581916    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b34b5d-79c7-4d32-b72d-9d6c398257a6" path="/var/lib/kubelet/pods/15b34b5d-79c7-4d32-b72d-9d6c398257a6/volumes"
Nov 02 13:55:37.264079 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:37.264044    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:55:41.132003 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:41.131072    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gt29z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:55:41.132003 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:41.131194    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podUID="bf09d973-3e5b-4daa-aa7f-cf839b61d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:55:50.646781 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.646741    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"]
Nov 02 13:55:50.647214 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.647186    4464 cpu_manager.go:414] "containerMap: removing container" podUID="985cb7db-44a3-4a5d-88d0-e02d0b98bc3e" containerName="payload-pre-processing"
Nov 02 13:55:50.647214 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.647199    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="985cb7db-44a3-4a5d-88d0-e02d0b98bc3e" containerName="payload-pre-processing"
Nov 02 13:55:50.647294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.647222    4464 cpu_manager.go:414] "containerMap: removing container" podUID="15b34b5d-79c7-4d32-b72d-9d6c398257a6" containerName="istio-proxy"
Nov 02 13:55:50.647294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.647227    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b34b5d-79c7-4d32-b72d-9d6c398257a6" containerName="istio-proxy"
Nov 02 13:55:50.647294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.647245    4464 cpu_manager.go:414] "containerMap: removing container" podUID="15b34b5d-79c7-4d32-b72d-9d6c398257a6" containerName="istio-proxy"
Nov 02 13:55:50.647294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.647252    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b34b5d-79c7-4d32-b72d-9d6c398257a6" containerName="istio-proxy"
Nov 02 13:55:50.647294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.647259    4464 cpu_manager.go:414] "containerMap: removing container" podUID="52d60cf1-1f4e-4436-9f0f-270fed53b0b4" containerName="payload-processing"
Nov 02 13:55:50.647294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.647263    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d60cf1-1f4e-4436-9f0f-270fed53b0b4" containerName="payload-processing"
Nov 02 13:55:50.647509 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.647322    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="985cb7db-44a3-4a5d-88d0-e02d0b98bc3e" containerName="payload-pre-processing"
Nov 02 13:55:50.647509 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.647333    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b34b5d-79c7-4d32-b72d-9d6c398257a6" containerName="istio-proxy"
Nov 02 13:55:50.647509 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.647340    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b34b5d-79c7-4d32-b72d-9d6c398257a6" containerName="istio-proxy"
Nov 02 13:55:50.647509 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.647346    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d60cf1-1f4e-4436-9f0f-270fed53b0b4" containerName="payload-processing"
Nov 02 13:55:50.650233 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.650216    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"
Nov 02 13:55:50.699148 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.699122    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhz27\" (UniqueName: \"kubernetes.io/projected/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-kube-api-access-zhz27\") pod \"maas-api-revoke-sub-0c62c279-f89tl\" (UID: \"f56d24d4-2246-43b9-b6fc-f18a73e91ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"
Nov 02 13:55:50.699273 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.699154    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c62c279-f89tl\" (UID: \"f56d24d4-2246-43b9-b6fc-f18a73e91ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"
Nov 02 13:55:50.699323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.699289    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c62c279-f89tl\" (UID: \"f56d24d4-2246-43b9-b6fc-f18a73e91ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"
Nov 02 13:55:50.708033 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.707959    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"]
Nov 02 13:55:50.756127 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.756102    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:55:50.756127 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.756101    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:55:50.756371 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.756354    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:55:50.756427 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.756382    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:55:50.799988 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.799943    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c62c279-f89tl\" (UID: \"f56d24d4-2246-43b9-b6fc-f18a73e91ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"
Nov 02 13:55:50.800145 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.800080    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhz27\" (UniqueName: \"kubernetes.io/projected/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-kube-api-access-zhz27\") pod \"maas-api-revoke-sub-0c62c279-f89tl\" (UID: \"f56d24d4-2246-43b9-b6fc-f18a73e91ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"
Nov 02 13:55:50.800145 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.800111    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c62c279-f89tl\" (UID: \"f56d24d4-2246-43b9-b6fc-f18a73e91ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"
Nov 02 13:55:50.800792 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.800773    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c62c279-f89tl\" (UID: \"f56d24d4-2246-43b9-b6fc-f18a73e91ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"
Nov 02 13:55:50.802316 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.802299    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c62c279-f89tl\" (UID: \"f56d24d4-2246-43b9-b6fc-f18a73e91ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"
Nov 02 13:55:50.862140 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.862110    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhz27\" (UniqueName: \"kubernetes.io/projected/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-kube-api-access-zhz27\") pod \"maas-api-revoke-sub-0c62c279-f89tl\" (UID: \"f56d24d4-2246-43b9-b6fc-f18a73e91ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"
Nov 02 13:55:50.962802 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:50.962732    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"
Nov 02 13:55:51.332669 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:51.332640    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"]
Nov 02 13:55:51.334598 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:55:51.334569    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56d24d4_2246_43b9_b6fc_f18a73e91ba9.slice/crio-cfd2e4b5f099399ad7b5da12da7a31ed6761ef9949c97833acca8b8bb4b0def7 WatchSource:0}: Error finding container cfd2e4b5f099399ad7b5da12da7a31ed6761ef9949c97833acca8b8bb4b0def7: Status 404 returned error can't find the container with id cfd2e4b5f099399ad7b5da12da7a31ed6761ef9949c97833acca8b8bb4b0def7
Nov 02 13:55:52.140665 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:52.140624    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl" event={"ID":"f56d24d4-2246-43b9-b6fc-f18a73e91ba9","Type":"ContainerStarted","Data":"e9a112d8e094d278cd45f13986570eede3b64df6fb9c7e6b8d207a10b8080e68"}
Nov 02 13:55:52.141073 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:52.140686    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl" event={"ID":"f56d24d4-2246-43b9-b6fc-f18a73e91ba9","Type":"ContainerStarted","Data":"cfd2e4b5f099399ad7b5da12da7a31ed6761ef9949c97833acca8b8bb4b0def7"}
Nov 02 13:55:52.259156 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:52.259106    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl" podStartSLOduration=2.25908927 podStartE2EDuration="2.25908927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:55:52.257881939 +0000 UTC m=+7180.821590350" watchObservedRunningTime="2026-11-02 13:55:52.25908927 +0000 UTC m=+7180.822797680"
Nov 02 13:55:53.146462 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:53.146429    4464 generic.go:356] "Generic (PLEG): container finished" podID="f56d24d4-2246-43b9-b6fc-f18a73e91ba9" containerID="e9a112d8e094d278cd45f13986570eede3b64df6fb9c7e6b8d207a10b8080e68" exitCode=0
Nov 02 13:55:53.146886 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:53.146515    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl" event={"ID":"f56d24d4-2246-43b9-b6fc-f18a73e91ba9","Type":"ContainerDied","Data":"e9a112d8e094d278cd45f13986570eede3b64df6fb9c7e6b8d207a10b8080e68"}
Nov 02 13:55:54.286655 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:54.286622    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"
Nov 02 13:55:54.331230 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:54.331208    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-kube-api-access-zhz27\" (UniqueName: \"kubernetes.io/projected/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-kube-api-access-zhz27\") pod \"f56d24d4-2246-43b9-b6fc-f18a73e91ba9\" (UID: \"f56d24d4-2246-43b9-b6fc-f18a73e91ba9\") "
Nov 02 13:55:54.331354 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:54.331297    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-maas-api-xks-ca\") pod \"f56d24d4-2246-43b9-b6fc-f18a73e91ba9\" (UID: \"f56d24d4-2246-43b9-b6fc-f18a73e91ba9\") "
Nov 02 13:55:54.331407 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:54.331359    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-maas-api-openshift-ca\") pod \"f56d24d4-2246-43b9-b6fc-f18a73e91ba9\" (UID: \"f56d24d4-2246-43b9-b6fc-f18a73e91ba9\") "
Nov 02 13:55:54.331726 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:54.331697    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-maas-api-openshift-ca" pod "f56d24d4-2246-43b9-b6fc-f18a73e91ba9" (UID: "f56d24d4-2246-43b9-b6fc-f18a73e91ba9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:55:54.333323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:54.333297    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-maas-api-xks-ca" pod "f56d24d4-2246-43b9-b6fc-f18a73e91ba9" (UID: "f56d24d4-2246-43b9-b6fc-f18a73e91ba9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:55:54.333401 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:54.333312    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-kube-api-access-zhz27" pod "f56d24d4-2246-43b9-b6fc-f18a73e91ba9" (UID: "f56d24d4-2246-43b9-b6fc-f18a73e91ba9"). InnerVolumeSpecName "kube-api-access-zhz27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:55:54.432652 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:54.432579    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhz27\" (UniqueName: \"kubernetes.io/projected/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-kube-api-access-zhz27\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:55:54.432652 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:54.432607    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:55:54.432652 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:54.432616    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f56d24d4-2246-43b9-b6fc-f18a73e91ba9-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:55:55.157026 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:55.156993    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"
Nov 02 13:55:55.157195 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:55.157022    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl" event={"ID":"f56d24d4-2246-43b9-b6fc-f18a73e91ba9","Type":"ContainerDied","Data":"cfd2e4b5f099399ad7b5da12da7a31ed6761ef9949c97833acca8b8bb4b0def7"}
Nov 02 13:55:55.157195 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:55:55.157074    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfd2e4b5f099399ad7b5da12da7a31ed6761ef9949c97833acca8b8bb4b0def7"
Nov 02 13:55:58.291142 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:55:58.291109    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:56:01.384327 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.384292    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"]
Nov 02 13:56:01.393422 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.385473    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f56d24d4-2246-43b9-b6fc-f18a73e91ba9" containerName="revoke-keys"
Nov 02 13:56:01.393422 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.385500    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f56d24d4-2246-43b9-b6fc-f18a73e91ba9" containerName="revoke-keys"
Nov 02 13:56:01.393422 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.385922    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f56d24d4-2246-43b9-b6fc-f18a73e91ba9" containerName="revoke-keys"
Nov 02 13:56:01.394771 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.394748    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"
Nov 02 13:56:01.446809 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.446773    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"]
Nov 02 13:56:01.450305 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.450266    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:56:01.495016 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.494960    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:56:01.495173 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.495034    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:56:01.495173 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.495082    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:56:01.500570 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.500543    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f1f567-3513-45e6-a1fd-0ae80e4e628d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f87662f-zt2bf\" (UID: \"32f1f567-3513-45e6-a1fd-0ae80e4e628d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"
Nov 02 13:56:01.500695 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.500598    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bf9p\" (UniqueName: \"kubernetes.io/projected/32f1f567-3513-45e6-a1fd-0ae80e4e628d-kube-api-access-5bf9p\") pod \"maas-api-revoke-sub-2f87662f-zt2bf\" (UID: \"32f1f567-3513-45e6-a1fd-0ae80e4e628d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"
Nov 02 13:56:01.500761 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.500744    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f1f567-3513-45e6-a1fd-0ae80e4e628d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f87662f-zt2bf\" (UID: \"32f1f567-3513-45e6-a1fd-0ae80e4e628d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"
Nov 02 13:56:01.601824 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.601799    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bf9p\" (UniqueName: \"kubernetes.io/projected/32f1f567-3513-45e6-a1fd-0ae80e4e628d-kube-api-access-5bf9p\") pod \"maas-api-revoke-sub-2f87662f-zt2bf\" (UID: \"32f1f567-3513-45e6-a1fd-0ae80e4e628d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"
Nov 02 13:56:01.601960 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.601891    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f1f567-3513-45e6-a1fd-0ae80e4e628d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f87662f-zt2bf\" (UID: \"32f1f567-3513-45e6-a1fd-0ae80e4e628d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"
Nov 02 13:56:01.601960 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.601952    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f1f567-3513-45e6-a1fd-0ae80e4e628d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f87662f-zt2bf\" (UID: \"32f1f567-3513-45e6-a1fd-0ae80e4e628d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"
Nov 02 13:56:01.602513 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.602494    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f1f567-3513-45e6-a1fd-0ae80e4e628d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f87662f-zt2bf\" (UID: \"32f1f567-3513-45e6-a1fd-0ae80e4e628d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"
Nov 02 13:56:01.604266 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.604241    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f1f567-3513-45e6-a1fd-0ae80e4e628d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f87662f-zt2bf\" (UID: \"32f1f567-3513-45e6-a1fd-0ae80e4e628d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"
Nov 02 13:56:01.663532 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.663468    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bf9p\" (UniqueName: \"kubernetes.io/projected/32f1f567-3513-45e6-a1fd-0ae80e4e628d-kube-api-access-5bf9p\") pod \"maas-api-revoke-sub-2f87662f-zt2bf\" (UID: \"32f1f567-3513-45e6-a1fd-0ae80e4e628d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"
Nov 02 13:56:01.708142 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:01.708116    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"
Nov 02 13:56:02.082265 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:02.082237    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"]
Nov 02 13:56:02.083630 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:56:02.083600    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f1f567_3513_45e6_a1fd_0ae80e4e628d.slice/crio-24cee4014f48677e92da9d1a5fc79a91ecfd3ef88bc332e0dcd8ae115a0933b5 WatchSource:0}: Error finding container 24cee4014f48677e92da9d1a5fc79a91ecfd3ef88bc332e0dcd8ae115a0933b5: Status 404 returned error can't find the container with id 24cee4014f48677e92da9d1a5fc79a91ecfd3ef88bc332e0dcd8ae115a0933b5
Nov 02 13:56:02.192569 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:02.192538    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf" event={"ID":"32f1f567-3513-45e6-a1fd-0ae80e4e628d","Type":"ContainerStarted","Data":"24cee4014f48677e92da9d1a5fc79a91ecfd3ef88bc332e0dcd8ae115a0933b5"}
Nov 02 13:56:03.198442 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:03.198357    4464 generic.go:356] "Generic (PLEG): container finished" podID="32f1f567-3513-45e6-a1fd-0ae80e4e628d" containerID="983ca65b69a98edbb54cbebd7ace80c86ef190d23f328a01cd4567217a2bf8f3" exitCode=0
Nov 02 13:56:03.198824 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:03.198438    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf" event={"ID":"32f1f567-3513-45e6-a1fd-0ae80e4e628d","Type":"ContainerDied","Data":"983ca65b69a98edbb54cbebd7ace80c86ef190d23f328a01cd4567217a2bf8f3"}
Nov 02 13:56:04.334161 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:04.334132    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"
Nov 02 13:56:04.429514 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:04.429487    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f1f567-3513-45e6-a1fd-0ae80e4e628d-kube-api-access-5bf9p\" (UniqueName: \"kubernetes.io/projected/32f1f567-3513-45e6-a1fd-0ae80e4e628d-kube-api-access-5bf9p\") pod \"32f1f567-3513-45e6-a1fd-0ae80e4e628d\" (UID: \"32f1f567-3513-45e6-a1fd-0ae80e4e628d\") "
Nov 02 13:56:04.429715 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:04.429547    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32f1f567-3513-45e6-a1fd-0ae80e4e628d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f1f567-3513-45e6-a1fd-0ae80e4e628d-maas-api-openshift-ca\") pod \"32f1f567-3513-45e6-a1fd-0ae80e4e628d\" (UID: \"32f1f567-3513-45e6-a1fd-0ae80e4e628d\") "
Nov 02 13:56:04.429715 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:04.429585    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32f1f567-3513-45e6-a1fd-0ae80e4e628d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f1f567-3513-45e6-a1fd-0ae80e4e628d-maas-api-xks-ca\") pod \"32f1f567-3513-45e6-a1fd-0ae80e4e628d\" (UID: \"32f1f567-3513-45e6-a1fd-0ae80e4e628d\") "
Nov 02 13:56:04.429915 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:04.429895    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f1f567-3513-45e6-a1fd-0ae80e4e628d-maas-api-openshift-ca" pod "32f1f567-3513-45e6-a1fd-0ae80e4e628d" (UID: "32f1f567-3513-45e6-a1fd-0ae80e4e628d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:04.431590 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:04.431558    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f1f567-3513-45e6-a1fd-0ae80e4e628d-kube-api-access-5bf9p" pod "32f1f567-3513-45e6-a1fd-0ae80e4e628d" (UID: "32f1f567-3513-45e6-a1fd-0ae80e4e628d"). InnerVolumeSpecName "kube-api-access-5bf9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:04.431704 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:04.431596    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f1f567-3513-45e6-a1fd-0ae80e4e628d-maas-api-xks-ca" pod "32f1f567-3513-45e6-a1fd-0ae80e4e628d" (UID: "32f1f567-3513-45e6-a1fd-0ae80e4e628d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:04.531257 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:04.531225    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f1f567-3513-45e6-a1fd-0ae80e4e628d-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:56:04.531257 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:04.531252    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bf9p\" (UniqueName: \"kubernetes.io/projected/32f1f567-3513-45e6-a1fd-0ae80e4e628d-kube-api-access-5bf9p\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:56:04.531257 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:04.531261    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f1f567-3513-45e6-a1fd-0ae80e4e628d-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:56:05.209623 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:05.209589    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"
Nov 02 13:56:05.209803 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:05.209618    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf" event={"ID":"32f1f567-3513-45e6-a1fd-0ae80e4e628d","Type":"ContainerDied","Data":"24cee4014f48677e92da9d1a5fc79a91ecfd3ef88bc332e0dcd8ae115a0933b5"}
Nov 02 13:56:05.209803 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:05.209666    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24cee4014f48677e92da9d1a5fc79a91ecfd3ef88bc332e0dcd8ae115a0933b5"
Nov 02 13:56:12.484059 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:12.484031    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:56:12.500629 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:12.500609    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:56:12.540287 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:12.540269    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 13:56:12.554780 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:12.554764    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 13:56:14.396843 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:14.396808    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:56:15.546710 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:15.546678    4464 scope.go:122] "RemoveContainer" containerID="cb0dcda686b670beaf37f2741230751a1894c34e84cdef894778db589c970c90"
Nov 02 13:56:23.069000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:23.063242    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"]
Nov 02 13:56:23.105997 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:23.105562    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2266010-d445x"]
Nov 02 13:56:23.694004 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:23.687472    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a14d2b91-407b-4843-af74-67b3298caf27" path="/var/lib/kubelet/pods/a14d2b91-407b-4843-af74-67b3298caf27/volumes"
Nov 02 13:56:25.612096 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:25.612056    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:56:28.292243 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:56:28.292204    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:56:36.893622 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:36.893588    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"]
Nov 02 13:56:36.894145 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:36.894041    4464 cpu_manager.go:414] "containerMap: removing container" podUID="32f1f567-3513-45e6-a1fd-0ae80e4e628d" containerName="revoke-keys"
Nov 02 13:56:36.894145 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:36.894053    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f1f567-3513-45e6-a1fd-0ae80e4e628d" containerName="revoke-keys"
Nov 02 13:56:36.894145 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:36.894132    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f1f567-3513-45e6-a1fd-0ae80e4e628d" containerName="revoke-keys"
Nov 02 13:56:36.898611 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:36.898594    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"
Nov 02 13:56:36.914845 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:36.914821    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf8gp\" (UniqueName: \"kubernetes.io/projected/6fc37496-e0cf-4914-abb3-8353a81d4b13-kube-api-access-jf8gp\") pod \"maas-api-revoke-sub-7e9bae84-x88jj\" (UID: \"6fc37496-e0cf-4914-abb3-8353a81d4b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"
Nov 02 13:56:36.915001 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:36.914871    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc37496-e0cf-4914-abb3-8353a81d4b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e9bae84-x88jj\" (UID: \"6fc37496-e0cf-4914-abb3-8353a81d4b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"
Nov 02 13:56:36.915107 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:36.915028    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc37496-e0cf-4914-abb3-8353a81d4b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e9bae84-x88jj\" (UID: \"6fc37496-e0cf-4914-abb3-8353a81d4b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"
Nov 02 13:56:36.952390 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:36.952362    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:56:37.003047 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:37.003023    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:56:37.003272 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:37.003255    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:56:37.003355 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:37.003275    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:56:37.005372 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:37.005347    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"]
Nov 02 13:56:37.015938 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:37.015914    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc37496-e0cf-4914-abb3-8353a81d4b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e9bae84-x88jj\" (UID: \"6fc37496-e0cf-4914-abb3-8353a81d4b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"
Nov 02 13:56:37.016058 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:37.016015    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf8gp\" (UniqueName: \"kubernetes.io/projected/6fc37496-e0cf-4914-abb3-8353a81d4b13-kube-api-access-jf8gp\") pod \"maas-api-revoke-sub-7e9bae84-x88jj\" (UID: \"6fc37496-e0cf-4914-abb3-8353a81d4b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"
Nov 02 13:56:37.016126 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:37.016086    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc37496-e0cf-4914-abb3-8353a81d4b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e9bae84-x88jj\" (UID: \"6fc37496-e0cf-4914-abb3-8353a81d4b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"
Nov 02 13:56:37.016533 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:37.016515    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc37496-e0cf-4914-abb3-8353a81d4b13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e9bae84-x88jj\" (UID: \"6fc37496-e0cf-4914-abb3-8353a81d4b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"
Nov 02 13:56:37.018255 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:37.018234    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc37496-e0cf-4914-abb3-8353a81d4b13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e9bae84-x88jj\" (UID: \"6fc37496-e0cf-4914-abb3-8353a81d4b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"
Nov 02 13:56:37.080659 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:37.080635    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf8gp\" (UniqueName: \"kubernetes.io/projected/6fc37496-e0cf-4914-abb3-8353a81d4b13-kube-api-access-jf8gp\") pod \"maas-api-revoke-sub-7e9bae84-x88jj\" (UID: \"6fc37496-e0cf-4914-abb3-8353a81d4b13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"
Nov 02 13:56:37.211743 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:37.211684    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"
Nov 02 13:56:37.582593 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:37.582565    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"]
Nov 02 13:56:37.584251 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:56:37.584225    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc37496_e0cf_4914_abb3_8353a81d4b13.slice/crio-51a4bfa9c73b8d9bbd88821b083047167e756a15944afb622884f677811c57d4 WatchSource:0}: Error finding container 51a4bfa9c73b8d9bbd88821b083047167e756a15944afb622884f677811c57d4: Status 404 returned error can't find the container with id 51a4bfa9c73b8d9bbd88821b083047167e756a15944afb622884f677811c57d4
Nov 02 13:56:37.665949 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:37.665918    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj" event={"ID":"6fc37496-e0cf-4914-abb3-8353a81d4b13","Type":"ContainerStarted","Data":"51a4bfa9c73b8d9bbd88821b083047167e756a15944afb622884f677811c57d4"}
Nov 02 13:56:38.672177 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:38.672145    4464 generic.go:356] "Generic (PLEG): container finished" podID="6fc37496-e0cf-4914-abb3-8353a81d4b13" containerID="9855d2984ba2c05c56f948fef20cd963e543f389f02d97c868349d3296f867ab" exitCode=0
Nov 02 13:56:38.672607 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:38.672231    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj" event={"ID":"6fc37496-e0cf-4914-abb3-8353a81d4b13","Type":"ContainerDied","Data":"9855d2984ba2c05c56f948fef20cd963e543f389f02d97c868349d3296f867ab"}
Nov 02 13:56:39.815950 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:39.815926    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"
Nov 02 13:56:39.842991 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:39.842958    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fc37496-e0cf-4914-abb3-8353a81d4b13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc37496-e0cf-4914-abb3-8353a81d4b13-maas-api-xks-ca\") pod \"6fc37496-e0cf-4914-abb3-8353a81d4b13\" (UID: \"6fc37496-e0cf-4914-abb3-8353a81d4b13\") "
Nov 02 13:56:39.843114 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:39.843064    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc37496-e0cf-4914-abb3-8353a81d4b13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc37496-e0cf-4914-abb3-8353a81d4b13-maas-api-openshift-ca\") pod \"6fc37496-e0cf-4914-abb3-8353a81d4b13\" (UID: \"6fc37496-e0cf-4914-abb3-8353a81d4b13\") "
Nov 02 13:56:39.843114 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:39.843107    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc37496-e0cf-4914-abb3-8353a81d4b13-kube-api-access-jf8gp\" (UniqueName: \"kubernetes.io/projected/6fc37496-e0cf-4914-abb3-8353a81d4b13-kube-api-access-jf8gp\") pod \"6fc37496-e0cf-4914-abb3-8353a81d4b13\" (UID: \"6fc37496-e0cf-4914-abb3-8353a81d4b13\") "
Nov 02 13:56:39.843441 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:39.843419    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc37496-e0cf-4914-abb3-8353a81d4b13-maas-api-openshift-ca" pod "6fc37496-e0cf-4914-abb3-8353a81d4b13" (UID: "6fc37496-e0cf-4914-abb3-8353a81d4b13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:39.843548 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:39.843532    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc37496-e0cf-4914-abb3-8353a81d4b13-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:56:39.844884 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:39.844855    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc37496-e0cf-4914-abb3-8353a81d4b13-maas-api-xks-ca" pod "6fc37496-e0cf-4914-abb3-8353a81d4b13" (UID: "6fc37496-e0cf-4914-abb3-8353a81d4b13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:39.845004 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:39.844951    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc37496-e0cf-4914-abb3-8353a81d4b13-kube-api-access-jf8gp" pod "6fc37496-e0cf-4914-abb3-8353a81d4b13" (UID: "6fc37496-e0cf-4914-abb3-8353a81d4b13"). InnerVolumeSpecName "kube-api-access-jf8gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:39.944923 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:39.944853    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc37496-e0cf-4914-abb3-8353a81d4b13-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:56:39.944923 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:39.944891    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf8gp\" (UniqueName: \"kubernetes.io/projected/6fc37496-e0cf-4914-abb3-8353a81d4b13-kube-api-access-jf8gp\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:56:40.684032 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:40.683993    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj" event={"ID":"6fc37496-e0cf-4914-abb3-8353a81d4b13","Type":"ContainerDied","Data":"51a4bfa9c73b8d9bbd88821b083047167e756a15944afb622884f677811c57d4"}
Nov 02 13:56:40.684207 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:40.684043    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51a4bfa9c73b8d9bbd88821b083047167e756a15944afb622884f677811c57d4"
Nov 02 13:56:40.684207 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:40.684005    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"
Nov 02 13:56:49.887241 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:56:49.887208    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:56:58.448079 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:56:58.446673    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:57:02.659222 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:02.659187    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:57:21.642476 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.642444    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"]
Nov 02 13:57:21.642945 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.642870    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6fc37496-e0cf-4914-abb3-8353a81d4b13" containerName="revoke-keys"
Nov 02 13:57:21.642945 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.642882    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc37496-e0cf-4914-abb3-8353a81d4b13" containerName="revoke-keys"
Nov 02 13:57:21.643063 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.642971    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc37496-e0cf-4914-abb3-8353a81d4b13" containerName="revoke-keys"
Nov 02 13:57:21.646261 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.646239    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"
Nov 02 13:57:21.656080 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.656052    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23005db-6dc2-4e98-b839-0963d0e5a8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc0b8ec5-p8pgn\" (UID: \"e23005db-6dc2-4e98-b839-0963d0e5a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"
Nov 02 13:57:21.656195 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.656107    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23005db-6dc2-4e98-b839-0963d0e5a8cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc0b8ec5-p8pgn\" (UID: \"e23005db-6dc2-4e98-b839-0963d0e5a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"
Nov 02 13:57:21.656241 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.656219    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqr4f\" (UniqueName: \"kubernetes.io/projected/e23005db-6dc2-4e98-b839-0963d0e5a8cd-kube-api-access-mqr4f\") pod \"maas-api-revoke-sub-cc0b8ec5-p8pgn\" (UID: \"e23005db-6dc2-4e98-b839-0963d0e5a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"
Nov 02 13:57:21.701020 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.700994    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:57:21.701212 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.701043    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:57:21.701212 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.701073    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:57:21.751790 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.751768    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:57:21.753922 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.753900    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"]
Nov 02 13:57:21.757590 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.757571    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23005db-6dc2-4e98-b839-0963d0e5a8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc0b8ec5-p8pgn\" (UID: \"e23005db-6dc2-4e98-b839-0963d0e5a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"
Nov 02 13:57:21.757668 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.757604    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23005db-6dc2-4e98-b839-0963d0e5a8cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc0b8ec5-p8pgn\" (UID: \"e23005db-6dc2-4e98-b839-0963d0e5a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"
Nov 02 13:57:21.757668 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.757623    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqr4f\" (UniqueName: \"kubernetes.io/projected/e23005db-6dc2-4e98-b839-0963d0e5a8cd-kube-api-access-mqr4f\") pod \"maas-api-revoke-sub-cc0b8ec5-p8pgn\" (UID: \"e23005db-6dc2-4e98-b839-0963d0e5a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"
Nov 02 13:57:21.758234 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.758213    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23005db-6dc2-4e98-b839-0963d0e5a8cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc0b8ec5-p8pgn\" (UID: \"e23005db-6dc2-4e98-b839-0963d0e5a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"
Nov 02 13:57:21.759802 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.759783    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23005db-6dc2-4e98-b839-0963d0e5a8cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc0b8ec5-p8pgn\" (UID: \"e23005db-6dc2-4e98-b839-0963d0e5a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"
Nov 02 13:57:21.819949 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.819924    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqr4f\" (UniqueName: \"kubernetes.io/projected/e23005db-6dc2-4e98-b839-0963d0e5a8cd-kube-api-access-mqr4f\") pod \"maas-api-revoke-sub-cc0b8ec5-p8pgn\" (UID: \"e23005db-6dc2-4e98-b839-0963d0e5a8cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"
Nov 02 13:57:21.958771 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:21.958693    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"
Nov 02 13:57:22.331275 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:22.331251    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"]
Nov 02 13:57:22.332999 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:57:22.332953    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23005db_6dc2_4e98_b839_0963d0e5a8cd.slice/crio-0a5bb2bc80056098eb1c3c0ce098c3710f591232b91c5431d5267cbc8c37d2ff WatchSource:0}: Error finding container 0a5bb2bc80056098eb1c3c0ce098c3710f591232b91c5431d5267cbc8c37d2ff: Status 404 returned error can't find the container with id 0a5bb2bc80056098eb1c3c0ce098c3710f591232b91c5431d5267cbc8c37d2ff
Nov 02 13:57:23.012425 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:23.012387    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn" event={"ID":"e23005db-6dc2-4e98-b839-0963d0e5a8cd","Type":"ContainerStarted","Data":"905c2c42cf512a2a7e75ce925e94ee017311589f805bf05e7ba057a25e4d63e9"}
Nov 02 13:57:23.012818 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:23.012494    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn" event={"ID":"e23005db-6dc2-4e98-b839-0963d0e5a8cd","Type":"ContainerStarted","Data":"0a5bb2bc80056098eb1c3c0ce098c3710f591232b91c5431d5267cbc8c37d2ff"}
Nov 02 13:57:23.131040 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:23.130992    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn" podStartSLOduration=2.130964632 podStartE2EDuration="2.130964632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:23.129768281 +0000 UTC m=+7271.693476690" watchObservedRunningTime="2026-11-02 13:57:23.130964632 +0000 UTC m=+7271.694673038"
Nov 02 13:57:24.019281 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:24.019246    4464 generic.go:356] "Generic (PLEG): container finished" podID="e23005db-6dc2-4e98-b839-0963d0e5a8cd" containerID="905c2c42cf512a2a7e75ce925e94ee017311589f805bf05e7ba057a25e4d63e9" exitCode=0
Nov 02 13:57:24.019681 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:24.019300    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn" event={"ID":"e23005db-6dc2-4e98-b839-0963d0e5a8cd","Type":"ContainerDied","Data":"905c2c42cf512a2a7e75ce925e94ee017311589f805bf05e7ba057a25e4d63e9"}
Nov 02 13:57:25.157056 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:25.157032    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"
Nov 02 13:57:25.188159 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:25.188125    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e23005db-6dc2-4e98-b839-0963d0e5a8cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23005db-6dc2-4e98-b839-0963d0e5a8cd-maas-api-xks-ca\") pod \"e23005db-6dc2-4e98-b839-0963d0e5a8cd\" (UID: \"e23005db-6dc2-4e98-b839-0963d0e5a8cd\") "
Nov 02 13:57:25.188309 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:25.188296    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23005db-6dc2-4e98-b839-0963d0e5a8cd-kube-api-access-mqr4f\" (UniqueName: \"kubernetes.io/projected/e23005db-6dc2-4e98-b839-0963d0e5a8cd-kube-api-access-mqr4f\") pod \"e23005db-6dc2-4e98-b839-0963d0e5a8cd\" (UID: \"e23005db-6dc2-4e98-b839-0963d0e5a8cd\") "
Nov 02 13:57:25.188361 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:25.188323    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23005db-6dc2-4e98-b839-0963d0e5a8cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23005db-6dc2-4e98-b839-0963d0e5a8cd-maas-api-openshift-ca\") pod \"e23005db-6dc2-4e98-b839-0963d0e5a8cd\" (UID: \"e23005db-6dc2-4e98-b839-0963d0e5a8cd\") "
Nov 02 13:57:25.188610 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:25.188589    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23005db-6dc2-4e98-b839-0963d0e5a8cd-maas-api-openshift-ca" pod "e23005db-6dc2-4e98-b839-0963d0e5a8cd" (UID: "e23005db-6dc2-4e98-b839-0963d0e5a8cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:57:25.188747 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:25.188704    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e23005db-6dc2-4e98-b839-0963d0e5a8cd-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:57:25.190143 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:25.190121    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e23005db-6dc2-4e98-b839-0963d0e5a8cd-maas-api-xks-ca" pod "e23005db-6dc2-4e98-b839-0963d0e5a8cd" (UID: "e23005db-6dc2-4e98-b839-0963d0e5a8cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:57:25.190229 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:25.190216    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23005db-6dc2-4e98-b839-0963d0e5a8cd-kube-api-access-mqr4f" pod "e23005db-6dc2-4e98-b839-0963d0e5a8cd" (UID: "e23005db-6dc2-4e98-b839-0963d0e5a8cd"). InnerVolumeSpecName "kube-api-access-mqr4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:57:25.289856 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:25.289764    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqr4f\" (UniqueName: \"kubernetes.io/projected/e23005db-6dc2-4e98-b839-0963d0e5a8cd-kube-api-access-mqr4f\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:57:25.289856 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:25.289804    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e23005db-6dc2-4e98-b839-0963d0e5a8cd-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:57:26.031785 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:26.031747    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn" event={"ID":"e23005db-6dc2-4e98-b839-0963d0e5a8cd","Type":"ContainerDied","Data":"0a5bb2bc80056098eb1c3c0ce098c3710f591232b91c5431d5267cbc8c37d2ff"}
Nov 02 13:57:26.031785 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:26.031778    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"
Nov 02 13:57:26.032044 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:26.031797    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a5bb2bc80056098eb1c3c0ce098c3710f591232b91c5431d5267cbc8c37d2ff"
Nov 02 13:57:28.290541 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:57:28.290502    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:57:32.426571 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.426537    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"]
Nov 02 13:57:32.427242 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.427222    4464 cpu_manager.go:414] "containerMap: removing container" podUID="e23005db-6dc2-4e98-b839-0963d0e5a8cd" containerName="revoke-keys"
Nov 02 13:57:32.427375 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.427244    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23005db-6dc2-4e98-b839-0963d0e5a8cd" containerName="revoke-keys"
Nov 02 13:57:32.427375 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.427335    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23005db-6dc2-4e98-b839-0963d0e5a8cd" containerName="revoke-keys"
Nov 02 13:57:32.430739 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.430711    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"
Nov 02 13:57:32.488791 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.488761    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"]
Nov 02 13:57:32.536545 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.536512    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:57:32.536717 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.536581    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:57:32.536717 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.536583    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:57:32.536717 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.536604    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:57:32.560199 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.560173    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93cafeac-cb97-4386-834b-a5602a4279dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e7fb344-nssdp\" (UID: \"93cafeac-cb97-4386-834b-a5602a4279dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"
Nov 02 13:57:32.560326 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.560312    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93cafeac-cb97-4386-834b-a5602a4279dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e7fb344-nssdp\" (UID: \"93cafeac-cb97-4386-834b-a5602a4279dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"
Nov 02 13:57:32.560377 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.560352    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkv2m\" (UniqueName: \"kubernetes.io/projected/93cafeac-cb97-4386-834b-a5602a4279dc-kube-api-access-gkv2m\") pod \"maas-api-revoke-sub-3e7fb344-nssdp\" (UID: \"93cafeac-cb97-4386-834b-a5602a4279dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"
Nov 02 13:57:32.661522 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.661490    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkv2m\" (UniqueName: \"kubernetes.io/projected/93cafeac-cb97-4386-834b-a5602a4279dc-kube-api-access-gkv2m\") pod \"maas-api-revoke-sub-3e7fb344-nssdp\" (UID: \"93cafeac-cb97-4386-834b-a5602a4279dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"
Nov 02 13:57:32.661703 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.661536    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93cafeac-cb97-4386-834b-a5602a4279dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e7fb344-nssdp\" (UID: \"93cafeac-cb97-4386-834b-a5602a4279dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"
Nov 02 13:57:32.661703 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.661611    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93cafeac-cb97-4386-834b-a5602a4279dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e7fb344-nssdp\" (UID: \"93cafeac-cb97-4386-834b-a5602a4279dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"
Nov 02 13:57:32.662225 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.662203    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93cafeac-cb97-4386-834b-a5602a4279dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e7fb344-nssdp\" (UID: \"93cafeac-cb97-4386-834b-a5602a4279dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"
Nov 02 13:57:32.663923 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.663900    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93cafeac-cb97-4386-834b-a5602a4279dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e7fb344-nssdp\" (UID: \"93cafeac-cb97-4386-834b-a5602a4279dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"
Nov 02 13:57:32.723567 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.723538    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkv2m\" (UniqueName: \"kubernetes.io/projected/93cafeac-cb97-4386-834b-a5602a4279dc-kube-api-access-gkv2m\") pod \"maas-api-revoke-sub-3e7fb344-nssdp\" (UID: \"93cafeac-cb97-4386-834b-a5602a4279dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"
Nov 02 13:57:32.743393 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:32.743372    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"
Nov 02 13:57:33.114630 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:33.114605    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"]
Nov 02 13:57:33.116070 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:57:33.116044    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93cafeac_cb97_4386_834b_a5602a4279dc.slice/crio-95c92a53cd833c6549813233d4341f7bcf76010781e714996ac6d2e923f6ce01 WatchSource:0}: Error finding container 95c92a53cd833c6549813233d4341f7bcf76010781e714996ac6d2e923f6ce01: Status 404 returned error can't find the container with id 95c92a53cd833c6549813233d4341f7bcf76010781e714996ac6d2e923f6ce01
Nov 02 13:57:34.074615 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:34.074519    4464 generic.go:356] "Generic (PLEG): container finished" podID="93cafeac-cb97-4386-834b-a5602a4279dc" containerID="c2db2d7f9bd13412c57df6eea168531c855d3b87b99979e353cf8b2758224c09" exitCode=0
Nov 02 13:57:34.075102 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:34.074612    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp" event={"ID":"93cafeac-cb97-4386-834b-a5602a4279dc","Type":"ContainerDied","Data":"c2db2d7f9bd13412c57df6eea168531c855d3b87b99979e353cf8b2758224c09"}
Nov 02 13:57:34.075102 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:34.074712    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp" event={"ID":"93cafeac-cb97-4386-834b-a5602a4279dc","Type":"ContainerStarted","Data":"95c92a53cd833c6549813233d4341f7bcf76010781e714996ac6d2e923f6ce01"}
Nov 02 13:57:34.595131 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:34.595099    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:57:35.213493 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:35.213467    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"
Nov 02 13:57:35.287204 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:35.287168    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93cafeac-cb97-4386-834b-a5602a4279dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93cafeac-cb97-4386-834b-a5602a4279dc-maas-api-xks-ca\") pod \"93cafeac-cb97-4386-834b-a5602a4279dc\" (UID: \"93cafeac-cb97-4386-834b-a5602a4279dc\") "
Nov 02 13:57:35.287400 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:35.287281    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93cafeac-cb97-4386-834b-a5602a4279dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93cafeac-cb97-4386-834b-a5602a4279dc-maas-api-openshift-ca\") pod \"93cafeac-cb97-4386-834b-a5602a4279dc\" (UID: \"93cafeac-cb97-4386-834b-a5602a4279dc\") "
Nov 02 13:57:35.287478 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:35.287427    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93cafeac-cb97-4386-834b-a5602a4279dc-kube-api-access-gkv2m\" (UniqueName: \"kubernetes.io/projected/93cafeac-cb97-4386-834b-a5602a4279dc-kube-api-access-gkv2m\") pod \"93cafeac-cb97-4386-834b-a5602a4279dc\" (UID: \"93cafeac-cb97-4386-834b-a5602a4279dc\") "
Nov 02 13:57:35.287655 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:35.287632    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93cafeac-cb97-4386-834b-a5602a4279dc-maas-api-openshift-ca" pod "93cafeac-cb97-4386-834b-a5602a4279dc" (UID: "93cafeac-cb97-4386-834b-a5602a4279dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:57:35.287802 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:35.287785    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93cafeac-cb97-4386-834b-a5602a4279dc-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:57:35.289375 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:35.289340    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93cafeac-cb97-4386-834b-a5602a4279dc-maas-api-xks-ca" pod "93cafeac-cb97-4386-834b-a5602a4279dc" (UID: "93cafeac-cb97-4386-834b-a5602a4279dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:57:35.289460 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:35.289393    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93cafeac-cb97-4386-834b-a5602a4279dc-kube-api-access-gkv2m" pod "93cafeac-cb97-4386-834b-a5602a4279dc" (UID: "93cafeac-cb97-4386-834b-a5602a4279dc"). InnerVolumeSpecName "kube-api-access-gkv2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:57:35.388777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:35.388705    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkv2m\" (UniqueName: \"kubernetes.io/projected/93cafeac-cb97-4386-834b-a5602a4279dc-kube-api-access-gkv2m\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:57:35.388777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:35.388732    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93cafeac-cb97-4386-834b-a5602a4279dc-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:57:36.085006 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:36.084963    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"
Nov 02 13:57:36.085181 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:36.085002    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp" event={"ID":"93cafeac-cb97-4386-834b-a5602a4279dc","Type":"ContainerDied","Data":"95c92a53cd833c6549813233d4341f7bcf76010781e714996ac6d2e923f6ce01"}
Nov 02 13:57:36.085181 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:36.085051    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95c92a53cd833c6549813233d4341f7bcf76010781e714996ac6d2e923f6ce01"
Nov 02 13:57:45.198520 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:45.198481    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:57:56.129881 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:56.129839    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gt29z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:57:56.130412 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:56.129928    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podUID="bf09d973-3e5b-4daa-aa7f-cf839b61d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:58.291626 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:57:58.291590    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:57:58.305112 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:57:58.305077    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:58:09.783847 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:09.783809    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:58:15.583826 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:15.583794    4464 scope.go:122] "RemoveContainer" containerID="bd1916845e8e13b3112c21bc277fc8dc2dc3a9e50990891bc2a1c0769665abd4"
Nov 02 13:58:20.256046 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.256005    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l"]
Nov 02 13:58:20.256583 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.256564    4464 cpu_manager.go:414] "containerMap: removing container" podUID="93cafeac-cb97-4386-834b-a5602a4279dc" containerName="revoke-keys"
Nov 02 13:58:20.256583 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.256584    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93cafeac-cb97-4386-834b-a5602a4279dc" containerName="revoke-keys"
Nov 02 13:58:20.256712 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.256671    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="93cafeac-cb97-4386-834b-a5602a4279dc" containerName="revoke-keys"
Nov 02 13:58:20.261007 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.260991    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l"
Nov 02 13:58:20.315017 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.314933    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:58:20.315017 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.314956    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:58:20.365410 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.365377    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:58:20.365577 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.365461    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:58:20.367657 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.367636    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l"]
Nov 02 13:58:20.416209 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.416174    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe60823-6047-45b5-bf35-5f31620d27b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c36f452-txs7l\" (UID: \"ffe60823-6047-45b5-bf35-5f31620d27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l"
Nov 02 13:58:20.416403 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.416384    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe60823-6047-45b5-bf35-5f31620d27b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c36f452-txs7l\" (UID: \"ffe60823-6047-45b5-bf35-5f31620d27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l"
Nov 02 13:58:20.416497 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.416426    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q2zk\" (UniqueName: \"kubernetes.io/projected/ffe60823-6047-45b5-bf35-5f31620d27b3-kube-api-access-6q2zk\") pod \"maas-api-revoke-sub-2c36f452-txs7l\" (UID: \"ffe60823-6047-45b5-bf35-5f31620d27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l"
Nov 02 13:58:20.517098 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.517016    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe60823-6047-45b5-bf35-5f31620d27b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c36f452-txs7l\" (UID: \"ffe60823-6047-45b5-bf35-5f31620d27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l"
Nov 02 13:58:20.517098 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.517053    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q2zk\" (UniqueName: \"kubernetes.io/projected/ffe60823-6047-45b5-bf35-5f31620d27b3-kube-api-access-6q2zk\") pod \"maas-api-revoke-sub-2c36f452-txs7l\" (UID: \"ffe60823-6047-45b5-bf35-5f31620d27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l"
Nov 02 13:58:20.517341 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.517178    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe60823-6047-45b5-bf35-5f31620d27b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c36f452-txs7l\" (UID: \"ffe60823-6047-45b5-bf35-5f31620d27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l"
Nov 02 13:58:20.517735 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.517711    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe60823-6047-45b5-bf35-5f31620d27b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c36f452-txs7l\" (UID: \"ffe60823-6047-45b5-bf35-5f31620d27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l"
Nov 02 13:58:20.519545 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.519519    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe60823-6047-45b5-bf35-5f31620d27b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c36f452-txs7l\" (UID: \"ffe60823-6047-45b5-bf35-5f31620d27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l"
Nov 02 13:58:20.579447 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.579412    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q2zk\" (UniqueName: \"kubernetes.io/projected/ffe60823-6047-45b5-bf35-5f31620d27b3-kube-api-access-6q2zk\") pod \"maas-api-revoke-sub-2c36f452-txs7l\" (UID: \"ffe60823-6047-45b5-bf35-5f31620d27b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l"
Nov 02 13:58:20.873714 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:20.873635    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l"
Nov 02 13:58:21.244566 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:21.244535    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l"]
Nov 02 13:58:21.246207 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:58:21.246181    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe60823_6047_45b5_bf35_5f31620d27b3.slice/crio-77ba83923f15005831d16a35b72d58036c55b7c80ea0b4e2290ac8a47eae42a5 WatchSource:0}: Error finding container 77ba83923f15005831d16a35b72d58036c55b7c80ea0b4e2290ac8a47eae42a5: Status 404 returned error can't find the container with id 77ba83923f15005831d16a35b72d58036c55b7c80ea0b4e2290ac8a47eae42a5
Nov 02 13:58:21.452751 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:21.452714    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l" event={"ID":"ffe60823-6047-45b5-bf35-5f31620d27b3","Type":"ContainerStarted","Data":"77ba83923f15005831d16a35b72d58036c55b7c80ea0b4e2290ac8a47eae42a5"}
Nov 02 13:58:22.458726 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:22.458645    4464 generic.go:356] "Generic (PLEG): container finished" podID="ffe60823-6047-45b5-bf35-5f31620d27b3" containerID="2548f0df0aa62ef0153bcc25b554bc6f738bf4d990ca8ad57b80de587b5e74f0" exitCode=0
Nov 02 13:58:22.459119 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:22.458731    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l" event={"ID":"ffe60823-6047-45b5-bf35-5f31620d27b3","Type":"ContainerDied","Data":"2548f0df0aa62ef0153bcc25b554bc6f738bf4d990ca8ad57b80de587b5e74f0"}
Nov 02 13:58:23.594530 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:23.594504    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l"
Nov 02 13:58:23.750482 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:23.750451    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe60823-6047-45b5-bf35-5f31620d27b3-kube-api-access-6q2zk\" (UniqueName: \"kubernetes.io/projected/ffe60823-6047-45b5-bf35-5f31620d27b3-kube-api-access-6q2zk\") pod \"ffe60823-6047-45b5-bf35-5f31620d27b3\" (UID: \"ffe60823-6047-45b5-bf35-5f31620d27b3\") "
Nov 02 13:58:23.750668 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:23.750494    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffe60823-6047-45b5-bf35-5f31620d27b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe60823-6047-45b5-bf35-5f31620d27b3-maas-api-xks-ca\") pod \"ffe60823-6047-45b5-bf35-5f31620d27b3\" (UID: \"ffe60823-6047-45b5-bf35-5f31620d27b3\") "
Nov 02 13:58:23.750668 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:23.750525    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe60823-6047-45b5-bf35-5f31620d27b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe60823-6047-45b5-bf35-5f31620d27b3-maas-api-openshift-ca\") pod \"ffe60823-6047-45b5-bf35-5f31620d27b3\" (UID: \"ffe60823-6047-45b5-bf35-5f31620d27b3\") "
Nov 02 13:58:23.750894 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:23.750873    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe60823-6047-45b5-bf35-5f31620d27b3-maas-api-openshift-ca" pod "ffe60823-6047-45b5-bf35-5f31620d27b3" (UID: "ffe60823-6047-45b5-bf35-5f31620d27b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:23.752656 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:23.752633    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffe60823-6047-45b5-bf35-5f31620d27b3-maas-api-xks-ca" pod "ffe60823-6047-45b5-bf35-5f31620d27b3" (UID: "ffe60823-6047-45b5-bf35-5f31620d27b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:58:23.752754 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:23.752657    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe60823-6047-45b5-bf35-5f31620d27b3-kube-api-access-6q2zk" pod "ffe60823-6047-45b5-bf35-5f31620d27b3" (UID: "ffe60823-6047-45b5-bf35-5f31620d27b3"). InnerVolumeSpecName "kube-api-access-6q2zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:58:23.851584 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:23.851555    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q2zk\" (UniqueName: \"kubernetes.io/projected/ffe60823-6047-45b5-bf35-5f31620d27b3-kube-api-access-6q2zk\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:58:23.851584 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:23.851580    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffe60823-6047-45b5-bf35-5f31620d27b3-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:58:23.851777 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:23.851593    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffe60823-6047-45b5-bf35-5f31620d27b3-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:58:24.469079 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:24.469049    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l"
Nov 02 13:58:24.469256 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:24.469095    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c36f452-txs7l" event={"ID":"ffe60823-6047-45b5-bf35-5f31620d27b3","Type":"ContainerDied","Data":"77ba83923f15005831d16a35b72d58036c55b7c80ea0b4e2290ac8a47eae42a5"}
Nov 02 13:58:24.469256 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:24.469147    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77ba83923f15005831d16a35b72d58036c55b7c80ea0b4e2290ac8a47eae42a5"
Nov 02 13:58:28.291404 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:58:28.291368    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:58:31.054328 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.054294    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz"]
Nov 02 13:58:31.056730 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.054927    4464 cpu_manager.go:414] "containerMap: removing container" podUID="ffe60823-6047-45b5-bf35-5f31620d27b3" containerName="revoke-keys"
Nov 02 13:58:31.056730 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.054943    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe60823-6047-45b5-bf35-5f31620d27b3" containerName="revoke-keys"
Nov 02 13:58:31.056730 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.055034    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe60823-6047-45b5-bf35-5f31620d27b3" containerName="revoke-keys"
Nov 02 13:58:31.058033 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.058011    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz"
Nov 02 13:58:31.112953 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.112923    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b69b23-2369-42cf-abd1-9103d400dba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39618c6d-czpvz\" (UID: \"f7b69b23-2369-42cf-abd1-9103d400dba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz"
Nov 02 13:58:31.113568 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.113541    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:58:31.113701 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.113647    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx2cj\" (UniqueName: \"kubernetes.io/projected/f7b69b23-2369-42cf-abd1-9103d400dba3-kube-api-access-zx2cj\") pod \"maas-api-revoke-sub-39618c6d-czpvz\" (UID: \"f7b69b23-2369-42cf-abd1-9103d400dba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz"
Nov 02 13:58:31.113859 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.113840    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b69b23-2369-42cf-abd1-9103d400dba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39618c6d-czpvz\" (UID: \"f7b69b23-2369-42cf-abd1-9103d400dba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz"
Nov 02 13:58:31.164014 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.163969    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:58:31.164014 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.163968    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:58:31.164268 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.164152    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:58:31.166388 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.166364    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz"]
Nov 02 13:58:31.214956 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.214925    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b69b23-2369-42cf-abd1-9103d400dba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39618c6d-czpvz\" (UID: \"f7b69b23-2369-42cf-abd1-9103d400dba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz"
Nov 02 13:58:31.215107 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.215016    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx2cj\" (UniqueName: \"kubernetes.io/projected/f7b69b23-2369-42cf-abd1-9103d400dba3-kube-api-access-zx2cj\") pod \"maas-api-revoke-sub-39618c6d-czpvz\" (UID: \"f7b69b23-2369-42cf-abd1-9103d400dba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz"
Nov 02 13:58:31.215107 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.215103    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b69b23-2369-42cf-abd1-9103d400dba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39618c6d-czpvz\" (UID: \"f7b69b23-2369-42cf-abd1-9103d400dba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz"
Nov 02 13:58:31.215682 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.215664    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b69b23-2369-42cf-abd1-9103d400dba3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39618c6d-czpvz\" (UID: \"f7b69b23-2369-42cf-abd1-9103d400dba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz"
Nov 02 13:58:31.217209 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.217187    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b69b23-2369-42cf-abd1-9103d400dba3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39618c6d-czpvz\" (UID: \"f7b69b23-2369-42cf-abd1-9103d400dba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz"
Nov 02 13:58:31.277265 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.277235    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx2cj\" (UniqueName: \"kubernetes.io/projected/f7b69b23-2369-42cf-abd1-9103d400dba3-kube-api-access-zx2cj\") pod \"maas-api-revoke-sub-39618c6d-czpvz\" (UID: \"f7b69b23-2369-42cf-abd1-9103d400dba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz"
Nov 02 13:58:31.372593 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.372526    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz"
Nov 02 13:58:31.746005 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:31.745486    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz"]
Nov 02 13:58:32.511855 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:32.511824    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz" event={"ID":"f7b69b23-2369-42cf-abd1-9103d400dba3","Type":"ContainerStarted","Data":"41938ffea2299aec9018d155128ff63c6c6d9d39d68d80cebe1d38d937b83c0f"}
Nov 02 13:58:32.512262 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:32.511879    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz" event={"ID":"f7b69b23-2369-42cf-abd1-9103d400dba3","Type":"ContainerStarted","Data":"acf57d320fcf205b1a563edbdca57d6564c8da3e7d41bdf2b283b9d5b4c60085"}
Nov 02 13:58:32.630794 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:32.630747    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz" podStartSLOduration=1.630734424 podStartE2EDuration="1.630734424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:32.629280737 +0000 UTC m=+7341.192989172" watchObservedRunningTime="2026-11-02 13:58:32.630734424 +0000 UTC m=+7341.194442843"
Nov 02 13:58:33.207566 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:33.207527    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:58:33.552997 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:33.552420    4464 generic.go:356] "Generic (PLEG): container finished" podID="f7b69b23-2369-42cf-abd1-9103d400dba3" containerID="41938ffea2299aec9018d155128ff63c6c6d9d39d68d80cebe1d38d937b83c0f" exitCode=0
Nov 02 13:58:33.552997 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:33.552672    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz" event={"ID":"f7b69b23-2369-42cf-abd1-9103d400dba3","Type":"ContainerDied","Data":"41938ffea2299aec9018d155128ff63c6c6d9d39d68d80cebe1d38d937b83c0f"}
Nov 02 13:58:34.760048 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:34.759760    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz"
Nov 02 13:58:34.884044 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:34.870296    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7b69b23-2369-42cf-abd1-9103d400dba3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b69b23-2369-42cf-abd1-9103d400dba3-maas-api-xks-ca\") pod \"f7b69b23-2369-42cf-abd1-9103d400dba3\" (UID: \"f7b69b23-2369-42cf-abd1-9103d400dba3\") "
Nov 02 13:58:34.884044 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:34.870448    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b69b23-2369-42cf-abd1-9103d400dba3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b69b23-2369-42cf-abd1-9103d400dba3-maas-api-openshift-ca\") pod \"f7b69b23-2369-42cf-abd1-9103d400dba3\" (UID: \"f7b69b23-2369-42cf-abd1-9103d400dba3\") "
Nov 02 13:58:34.884044 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:34.870486    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b69b23-2369-42cf-abd1-9103d400dba3-kube-api-access-zx2cj\" (UniqueName: \"kubernetes.io/projected/f7b69b23-2369-42cf-abd1-9103d400dba3-kube-api-access-zx2cj\") pod \"f7b69b23-2369-42cf-abd1-9103d400dba3\" (UID: \"f7b69b23-2369-42cf-abd1-9103d400dba3\") "
Nov 02 13:58:34.884044 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:34.871826    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b69b23-2369-42cf-abd1-9103d400dba3-maas-api-openshift-ca" pod "f7b69b23-2369-42cf-abd1-9103d400dba3" (UID: "f7b69b23-2369-42cf-abd1-9103d400dba3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:34.917161 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:34.917114    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7b69b23-2369-42cf-abd1-9103d400dba3-maas-api-xks-ca" pod "f7b69b23-2369-42cf-abd1-9103d400dba3" (UID: "f7b69b23-2369-42cf-abd1-9103d400dba3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:58:34.924170 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:34.924117    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b69b23-2369-42cf-abd1-9103d400dba3-kube-api-access-zx2cj" pod "f7b69b23-2369-42cf-abd1-9103d400dba3" (UID: "f7b69b23-2369-42cf-abd1-9103d400dba3"). InnerVolumeSpecName "kube-api-access-zx2cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:58:34.979005 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:34.971637    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b69b23-2369-42cf-abd1-9103d400dba3-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:58:34.979005 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:34.971674    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b69b23-2369-42cf-abd1-9103d400dba3-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:58:34.979005 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:34.971691    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx2cj\" (UniqueName: \"kubernetes.io/projected/f7b69b23-2369-42cf-abd1-9103d400dba3-kube-api-access-zx2cj\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:58:35.591098 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:35.591059    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz" event={"ID":"f7b69b23-2369-42cf-abd1-9103d400dba3","Type":"ContainerDied","Data":"acf57d320fcf205b1a563edbdca57d6564c8da3e7d41bdf2b283b9d5b4c60085"}
Nov 02 13:58:35.591386 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:35.591368    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acf57d320fcf205b1a563edbdca57d6564c8da3e7d41bdf2b283b9d5b4c60085"
Nov 02 13:58:35.591563 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:35.591550    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39618c6d-czpvz"
Nov 02 13:58:43.883073 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:43.883037    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:58:49.892007 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:49.891052    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dghm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:58:49.892007 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:49.891115    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7" podUID="2d10b658-29b5-4e07-84df-13326972bead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:50.042819 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:50.042079    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cnggl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:58:50.042819 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:50.042143    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl" podUID="91491db1-c3aa-422e-9e3d-5575b4197aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:52.179218 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:58:52.179184    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:58:58.292080 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:58:58.292044    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:59:13.281014 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:13.280963    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:59:21.861272 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:21.861234    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp"]
Nov 02 13:59:21.861763 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:21.861700    4464 cpu_manager.go:414] "containerMap: removing container" podUID="f7b69b23-2369-42cf-abd1-9103d400dba3" containerName="revoke-keys"
Nov 02 13:59:21.861763 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:21.861714    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b69b23-2369-42cf-abd1-9103d400dba3" containerName="revoke-keys"
Nov 02 13:59:21.861898 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:21.861800    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b69b23-2369-42cf-abd1-9103d400dba3" containerName="revoke-keys"
Nov 02 13:59:21.864958 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:21.864939    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp"
Nov 02 13:59:21.919867 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:21.919828    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:59:21.922774 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:21.922747    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp"]
Nov 02 13:59:21.966763 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:21.966736    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlcw6\" (UniqueName: \"kubernetes.io/projected/38ac09da-26a6-44f0-8d1e-a72581e3c468-kube-api-access-xlcw6\") pod \"maas-api-revoke-sub-8ba75117-r5sgp\" (UID: \"38ac09da-26a6-44f0-8d1e-a72581e3c468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp"
Nov 02 13:59:21.966923 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:21.966802    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ac09da-26a6-44f0-8d1e-a72581e3c468-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba75117-r5sgp\" (UID: \"38ac09da-26a6-44f0-8d1e-a72581e3c468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp"
Nov 02 13:59:21.966923 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:21.966890    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ac09da-26a6-44f0-8d1e-a72581e3c468-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba75117-r5sgp\" (UID: \"38ac09da-26a6-44f0-8d1e-a72581e3c468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp"
Nov 02 13:59:21.970254 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:21.970234    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:59:21.970363 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:21.970302    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:59:21.970363 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:21.970331    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:59:22.068057 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:22.068019    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlcw6\" (UniqueName: \"kubernetes.io/projected/38ac09da-26a6-44f0-8d1e-a72581e3c468-kube-api-access-xlcw6\") pod \"maas-api-revoke-sub-8ba75117-r5sgp\" (UID: \"38ac09da-26a6-44f0-8d1e-a72581e3c468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp"
Nov 02 13:59:22.068255 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:22.068091    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ac09da-26a6-44f0-8d1e-a72581e3c468-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba75117-r5sgp\" (UID: \"38ac09da-26a6-44f0-8d1e-a72581e3c468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp"
Nov 02 13:59:22.068255 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:22.068111    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ac09da-26a6-44f0-8d1e-a72581e3c468-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba75117-r5sgp\" (UID: \"38ac09da-26a6-44f0-8d1e-a72581e3c468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp"
Nov 02 13:59:22.068788 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:22.068769    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ac09da-26a6-44f0-8d1e-a72581e3c468-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba75117-r5sgp\" (UID: \"38ac09da-26a6-44f0-8d1e-a72581e3c468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp"
Nov 02 13:59:22.070292 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:22.070272    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ac09da-26a6-44f0-8d1e-a72581e3c468-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba75117-r5sgp\" (UID: \"38ac09da-26a6-44f0-8d1e-a72581e3c468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp"
Nov 02 13:59:22.130764 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:22.130699    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlcw6\" (UniqueName: \"kubernetes.io/projected/38ac09da-26a6-44f0-8d1e-a72581e3c468-kube-api-access-xlcw6\") pod \"maas-api-revoke-sub-8ba75117-r5sgp\" (UID: \"38ac09da-26a6-44f0-8d1e-a72581e3c468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp"
Nov 02 13:59:22.177559 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:22.177535    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp"
Nov 02 13:59:22.546034 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:22.546008    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp"]
Nov 02 13:59:22.547421 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:59:22.547394    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ac09da_26a6_44f0_8d1e_a72581e3c468.slice/crio-bf34cfccd3e8b9ddd67dd9f8f0624fdaf26155dd5aee1c13d31d054676de3a98 WatchSource:0}: Error finding container bf34cfccd3e8b9ddd67dd9f8f0624fdaf26155dd5aee1c13d31d054676de3a98: Status 404 returned error can't find the container with id bf34cfccd3e8b9ddd67dd9f8f0624fdaf26155dd5aee1c13d31d054676de3a98
Nov 02 13:59:23.179731 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:23.179645    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp" event={"ID":"38ac09da-26a6-44f0-8d1e-a72581e3c468","Type":"ContainerStarted","Data":"ec4861c53c8562e368187c85d802960f1447d7a1caf8c3511860f5c7065b15a1"}
Nov 02 13:59:23.179731 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:23.179700    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp" event={"ID":"38ac09da-26a6-44f0-8d1e-a72581e3c468","Type":"ContainerStarted","Data":"bf34cfccd3e8b9ddd67dd9f8f0624fdaf26155dd5aee1c13d31d054676de3a98"}
Nov 02 13:59:23.299245 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:23.299188    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp" podStartSLOduration=2.299174506 podStartE2EDuration="2.299174506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:23.298382076 +0000 UTC m=+7391.862090505" watchObservedRunningTime="2026-11-02 13:59:23.299174506 +0000 UTC m=+7391.862882914"
Nov 02 13:59:24.186090 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:24.186057    4464 generic.go:356] "Generic (PLEG): container finished" podID="38ac09da-26a6-44f0-8d1e-a72581e3c468" containerID="ec4861c53c8562e368187c85d802960f1447d7a1caf8c3511860f5c7065b15a1" exitCode=0
Nov 02 13:59:24.186511 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:24.186144    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp" event={"ID":"38ac09da-26a6-44f0-8d1e-a72581e3c468","Type":"ContainerDied","Data":"ec4861c53c8562e368187c85d802960f1447d7a1caf8c3511860f5c7065b15a1"}
Nov 02 13:59:25.326326 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:25.326302    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp"
Nov 02 13:59:25.398580 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:25.398552    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38ac09da-26a6-44f0-8d1e-a72581e3c468-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ac09da-26a6-44f0-8d1e-a72581e3c468-maas-api-xks-ca\") pod \"38ac09da-26a6-44f0-8d1e-a72581e3c468\" (UID: \"38ac09da-26a6-44f0-8d1e-a72581e3c468\") "
Nov 02 13:59:25.398751 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:25.398710    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ac09da-26a6-44f0-8d1e-a72581e3c468-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ac09da-26a6-44f0-8d1e-a72581e3c468-maas-api-openshift-ca\") pod \"38ac09da-26a6-44f0-8d1e-a72581e3c468\" (UID: \"38ac09da-26a6-44f0-8d1e-a72581e3c468\") "
Nov 02 13:59:25.398751 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:25.398743    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ac09da-26a6-44f0-8d1e-a72581e3c468-kube-api-access-xlcw6\" (UniqueName: \"kubernetes.io/projected/38ac09da-26a6-44f0-8d1e-a72581e3c468-kube-api-access-xlcw6\") pod \"38ac09da-26a6-44f0-8d1e-a72581e3c468\" (UID: \"38ac09da-26a6-44f0-8d1e-a72581e3c468\") "
Nov 02 13:59:25.399072 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:25.399050    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ac09da-26a6-44f0-8d1e-a72581e3c468-maas-api-openshift-ca" pod "38ac09da-26a6-44f0-8d1e-a72581e3c468" (UID: "38ac09da-26a6-44f0-8d1e-a72581e3c468"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:25.400568 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:25.400542    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ac09da-26a6-44f0-8d1e-a72581e3c468-maas-api-xks-ca" pod "38ac09da-26a6-44f0-8d1e-a72581e3c468" (UID: "38ac09da-26a6-44f0-8d1e-a72581e3c468"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:25.400799 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:25.400786    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ac09da-26a6-44f0-8d1e-a72581e3c468-kube-api-access-xlcw6" pod "38ac09da-26a6-44f0-8d1e-a72581e3c468" (UID: "38ac09da-26a6-44f0-8d1e-a72581e3c468"). InnerVolumeSpecName "kube-api-access-xlcw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:25.499924 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:25.499890    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ac09da-26a6-44f0-8d1e-a72581e3c468-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:59:25.499924 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:25.499920    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlcw6\" (UniqueName: \"kubernetes.io/projected/38ac09da-26a6-44f0-8d1e-a72581e3c468-kube-api-access-xlcw6\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:59:25.500152 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:25.499934    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ac09da-26a6-44f0-8d1e-a72581e3c468-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:59:26.203223 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:26.203189    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp"
Nov 02 13:59:26.203432 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:26.203231    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba75117-r5sgp" event={"ID":"38ac09da-26a6-44f0-8d1e-a72581e3c468","Type":"ContainerDied","Data":"bf34cfccd3e8b9ddd67dd9f8f0624fdaf26155dd5aee1c13d31d054676de3a98"}
Nov 02 13:59:26.203432 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:26.203303    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf34cfccd3e8b9ddd67dd9f8f0624fdaf26155dd5aee1c13d31d054676de3a98"
Nov 02 13:59:28.291097 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:59:28.291067    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:59:33.075039 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.075004    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv"]
Nov 02 13:59:33.075468 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.075454    4464 cpu_manager.go:414] "containerMap: removing container" podUID="38ac09da-26a6-44f0-8d1e-a72581e3c468" containerName="revoke-keys"
Nov 02 13:59:33.075514 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.075468    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ac09da-26a6-44f0-8d1e-a72581e3c468" containerName="revoke-keys"
Nov 02 13:59:33.075551 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.075542    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ac09da-26a6-44f0-8d1e-a72581e3c468" containerName="revoke-keys"
Nov 02 13:59:33.078809 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.078787    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv"
Nov 02 13:59:33.133884 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.133848    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:59:33.134090 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.133905    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:59:33.181921 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.181883    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacadd1d-2562-47c4-9cde-ce2c1066ca49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfdf614b-g47mv\" (UID: \"aacadd1d-2562-47c4-9cde-ce2c1066ca49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv"
Nov 02 13:59:33.182115 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.181928    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz888\" (UniqueName: \"kubernetes.io/projected/aacadd1d-2562-47c4-9cde-ce2c1066ca49-kube-api-access-cz888\") pod \"maas-api-revoke-sub-cfdf614b-g47mv\" (UID: \"aacadd1d-2562-47c4-9cde-ce2c1066ca49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv"
Nov 02 13:59:33.182115 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.182101    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacadd1d-2562-47c4-9cde-ce2c1066ca49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfdf614b-g47mv\" (UID: \"aacadd1d-2562-47c4-9cde-ce2c1066ca49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv"
Nov 02 13:59:33.184475 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.184453    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:59:33.184605 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.184498    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:59:33.186943 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.186926    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv"]
Nov 02 13:59:33.282862 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.282830    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacadd1d-2562-47c4-9cde-ce2c1066ca49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfdf614b-g47mv\" (UID: \"aacadd1d-2562-47c4-9cde-ce2c1066ca49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv"
Nov 02 13:59:33.283060 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.282916    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacadd1d-2562-47c4-9cde-ce2c1066ca49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfdf614b-g47mv\" (UID: \"aacadd1d-2562-47c4-9cde-ce2c1066ca49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv"
Nov 02 13:59:33.283060 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.282936    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz888\" (UniqueName: \"kubernetes.io/projected/aacadd1d-2562-47c4-9cde-ce2c1066ca49-kube-api-access-cz888\") pod \"maas-api-revoke-sub-cfdf614b-g47mv\" (UID: \"aacadd1d-2562-47c4-9cde-ce2c1066ca49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv"
Nov 02 13:59:33.283521 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.283495    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacadd1d-2562-47c4-9cde-ce2c1066ca49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfdf614b-g47mv\" (UID: \"aacadd1d-2562-47c4-9cde-ce2c1066ca49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv"
Nov 02 13:59:33.285139 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.285115    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacadd1d-2562-47c4-9cde-ce2c1066ca49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfdf614b-g47mv\" (UID: \"aacadd1d-2562-47c4-9cde-ce2c1066ca49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv"
Nov 02 13:59:33.344918 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.344856    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz888\" (UniqueName: \"kubernetes.io/projected/aacadd1d-2562-47c4-9cde-ce2c1066ca49-kube-api-access-cz888\") pod \"maas-api-revoke-sub-cfdf614b-g47mv\" (UID: \"aacadd1d-2562-47c4-9cde-ce2c1066ca49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv"
Nov 02 13:59:33.391893 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.391871    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv"
Nov 02 13:59:33.763132 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:33.763102    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv"]
Nov 02 13:59:33.764861 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:59:33.764833    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacadd1d_2562_47c4_9cde_ce2c1066ca49.slice/crio-93447ed5de35a7a8eda0d9e1a33e529f520aec203dbad7c77a2ab24a5df9f236 WatchSource:0}: Error finding container 93447ed5de35a7a8eda0d9e1a33e529f520aec203dbad7c77a2ab24a5df9f236: Status 404 returned error can't find the container with id 93447ed5de35a7a8eda0d9e1a33e529f520aec203dbad7c77a2ab24a5df9f236
Nov 02 13:59:34.251283 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:34.251250    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv" event={"ID":"aacadd1d-2562-47c4-9cde-ce2c1066ca49","Type":"ContainerStarted","Data":"e6fc8f659ca5376d42d8af8aef3fb93e1815a719d5b0398deaaa44307e5b1e8d"}
Nov 02 13:59:34.251669 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:34.251308    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv" event={"ID":"aacadd1d-2562-47c4-9cde-ce2c1066ca49","Type":"ContainerStarted","Data":"93447ed5de35a7a8eda0d9e1a33e529f520aec203dbad7c77a2ab24a5df9f236"}
Nov 02 13:59:34.369778 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:34.369683    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv" podStartSLOduration=1.369669668 podStartE2EDuration="1.369669668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:34.367500999 +0000 UTC m=+7402.931209409" watchObservedRunningTime="2026-11-02 13:59:34.369669668 +0000 UTC m=+7402.933378076"
Nov 02 13:59:35.257365 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:35.257329    4464 generic.go:356] "Generic (PLEG): container finished" podID="aacadd1d-2562-47c4-9cde-ce2c1066ca49" containerID="e6fc8f659ca5376d42d8af8aef3fb93e1815a719d5b0398deaaa44307e5b1e8d" exitCode=0
Nov 02 13:59:35.257751 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:35.257399    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv" event={"ID":"aacadd1d-2562-47c4-9cde-ce2c1066ca49","Type":"ContainerDied","Data":"e6fc8f659ca5376d42d8af8aef3fb93e1815a719d5b0398deaaa44307e5b1e8d"}
Nov 02 13:59:36.395755 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:36.395729    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv"
Nov 02 13:59:36.514558 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:36.513821    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aacadd1d-2562-47c4-9cde-ce2c1066ca49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacadd1d-2562-47c4-9cde-ce2c1066ca49-maas-api-openshift-ca\") pod \"aacadd1d-2562-47c4-9cde-ce2c1066ca49\" (UID: \"aacadd1d-2562-47c4-9cde-ce2c1066ca49\") "
Nov 02 13:59:36.514558 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:36.513880    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacadd1d-2562-47c4-9cde-ce2c1066ca49-kube-api-access-cz888\" (UniqueName: \"kubernetes.io/projected/aacadd1d-2562-47c4-9cde-ce2c1066ca49-kube-api-access-cz888\") pod \"aacadd1d-2562-47c4-9cde-ce2c1066ca49\" (UID: \"aacadd1d-2562-47c4-9cde-ce2c1066ca49\") "
Nov 02 13:59:36.514558 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:36.514068    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aacadd1d-2562-47c4-9cde-ce2c1066ca49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacadd1d-2562-47c4-9cde-ce2c1066ca49-maas-api-xks-ca\") pod \"aacadd1d-2562-47c4-9cde-ce2c1066ca49\" (UID: \"aacadd1d-2562-47c4-9cde-ce2c1066ca49\") "
Nov 02 13:59:36.514558 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:36.514487    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacadd1d-2562-47c4-9cde-ce2c1066ca49-maas-api-openshift-ca" pod "aacadd1d-2562-47c4-9cde-ce2c1066ca49" (UID: "aacadd1d-2562-47c4-9cde-ce2c1066ca49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:36.516383 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:36.516356    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aacadd1d-2562-47c4-9cde-ce2c1066ca49-maas-api-xks-ca" pod "aacadd1d-2562-47c4-9cde-ce2c1066ca49" (UID: "aacadd1d-2562-47c4-9cde-ce2c1066ca49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:36.516688 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:36.516669    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacadd1d-2562-47c4-9cde-ce2c1066ca49-kube-api-access-cz888" pod "aacadd1d-2562-47c4-9cde-ce2c1066ca49" (UID: "aacadd1d-2562-47c4-9cde-ce2c1066ca49"). InnerVolumeSpecName "kube-api-access-cz888". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:36.615294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:36.615261    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacadd1d-2562-47c4-9cde-ce2c1066ca49-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:59:36.615294 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:36.615293    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacadd1d-2562-47c4-9cde-ce2c1066ca49-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:59:36.615479 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:36.615303    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz888\" (UniqueName: \"kubernetes.io/projected/aacadd1d-2562-47c4-9cde-ce2c1066ca49-kube-api-access-cz888\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:59:36.671738 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:36.671708    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 13:59:37.272715 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:37.272672    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv" event={"ID":"aacadd1d-2562-47c4-9cde-ce2c1066ca49","Type":"ContainerDied","Data":"93447ed5de35a7a8eda0d9e1a33e529f520aec203dbad7c77a2ab24a5df9f236"}
Nov 02 13:59:37.272715 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:37.272704    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfdf614b-g47mv"
Nov 02 13:59:37.273036 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:37.272736    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93447ed5de35a7a8eda0d9e1a33e529f520aec203dbad7c77a2ab24a5df9f236"
Nov 02 13:59:43.789518 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:43.789482    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj"]
Nov 02 13:59:43.790084 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:43.790064    4464 cpu_manager.go:414] "containerMap: removing container" podUID="aacadd1d-2562-47c4-9cde-ce2c1066ca49" containerName="revoke-keys"
Nov 02 13:59:43.790084 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:43.790083    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aacadd1d-2562-47c4-9cde-ce2c1066ca49" containerName="revoke-keys"
Nov 02 13:59:43.790224 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:43.790175    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="aacadd1d-2562-47c4-9cde-ce2c1066ca49" containerName="revoke-keys"
Nov 02 13:59:43.793938 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:43.793901    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj"
Nov 02 13:59:43.898727 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:43.898692    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:59:43.898727 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:43.898692    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:59:43.899009 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:43.898732    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:59:43.899079 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:43.899060    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:59:43.901211 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:43.901190    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj"]
Nov 02 13:59:43.942706 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:43.942681    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df181c99-e72f-4b8e-9a08-e416dbfb8837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db4acbc8-bvpmj\" (UID: \"df181c99-e72f-4b8e-9a08-e416dbfb8837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj"
Nov 02 13:59:43.942846 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:43.942753    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df181c99-e72f-4b8e-9a08-e416dbfb8837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db4acbc8-bvpmj\" (UID: \"df181c99-e72f-4b8e-9a08-e416dbfb8837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj"
Nov 02 13:59:43.942945 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:43.942858    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s88k\" (UniqueName: \"kubernetes.io/projected/df181c99-e72f-4b8e-9a08-e416dbfb8837-kube-api-access-7s88k\") pod \"maas-api-revoke-sub-db4acbc8-bvpmj\" (UID: \"df181c99-e72f-4b8e-9a08-e416dbfb8837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj"
Nov 02 13:59:44.043607 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:44.043527    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df181c99-e72f-4b8e-9a08-e416dbfb8837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db4acbc8-bvpmj\" (UID: \"df181c99-e72f-4b8e-9a08-e416dbfb8837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj"
Nov 02 13:59:44.043607 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:44.043572    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s88k\" (UniqueName: \"kubernetes.io/projected/df181c99-e72f-4b8e-9a08-e416dbfb8837-kube-api-access-7s88k\") pod \"maas-api-revoke-sub-db4acbc8-bvpmj\" (UID: \"df181c99-e72f-4b8e-9a08-e416dbfb8837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj"
Nov 02 13:59:44.043806 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:44.043623    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df181c99-e72f-4b8e-9a08-e416dbfb8837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db4acbc8-bvpmj\" (UID: \"df181c99-e72f-4b8e-9a08-e416dbfb8837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj"
Nov 02 13:59:44.044283 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:44.044264    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df181c99-e72f-4b8e-9a08-e416dbfb8837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db4acbc8-bvpmj\" (UID: \"df181c99-e72f-4b8e-9a08-e416dbfb8837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj"
Nov 02 13:59:44.045920 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:44.045900    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df181c99-e72f-4b8e-9a08-e416dbfb8837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db4acbc8-bvpmj\" (UID: \"df181c99-e72f-4b8e-9a08-e416dbfb8837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj"
Nov 02 13:59:44.106174 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:44.106142    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s88k\" (UniqueName: \"kubernetes.io/projected/df181c99-e72f-4b8e-9a08-e416dbfb8837-kube-api-access-7s88k\") pod \"maas-api-revoke-sub-db4acbc8-bvpmj\" (UID: \"df181c99-e72f-4b8e-9a08-e416dbfb8837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj"
Nov 02 13:59:44.106900 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:44.106882    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj"
Nov 02 13:59:44.480552 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:44.480523    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj"]
Nov 02 13:59:44.481713 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:59:44.481684    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf181c99_e72f_4b8e_9a08_e416dbfb8837.slice/crio-960a50e7ba16015250ffd7315b2ad68eb02f9133e6442e2d1d1b9a5cb20f7375 WatchSource:0}: Error finding container 960a50e7ba16015250ffd7315b2ad68eb02f9133e6442e2d1d1b9a5cb20f7375: Status 404 returned error can't find the container with id 960a50e7ba16015250ffd7315b2ad68eb02f9133e6442e2d1d1b9a5cb20f7375
Nov 02 13:59:45.349567 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:45.349535    4464 generic.go:356] "Generic (PLEG): container finished" podID="df181c99-e72f-4b8e-9a08-e416dbfb8837" containerID="9ed533fe511f34c3282fef4445780a02f8a347557fcc13ad8b81defc5ceca163" exitCode=0
Nov 02 13:59:45.350012 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:45.349575    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj" event={"ID":"df181c99-e72f-4b8e-9a08-e416dbfb8837","Type":"ContainerDied","Data":"9ed533fe511f34c3282fef4445780a02f8a347557fcc13ad8b81defc5ceca163"}
Nov 02 13:59:45.350012 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:45.349663    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj" event={"ID":"df181c99-e72f-4b8e-9a08-e416dbfb8837","Type":"ContainerStarted","Data":"960a50e7ba16015250ffd7315b2ad68eb02f9133e6442e2d1d1b9a5cb20f7375"}
Nov 02 13:59:46.484376 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:46.484351    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj"
Nov 02 13:59:46.668274 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:46.668197    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df181c99-e72f-4b8e-9a08-e416dbfb8837-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df181c99-e72f-4b8e-9a08-e416dbfb8837-maas-api-openshift-ca\") pod \"df181c99-e72f-4b8e-9a08-e416dbfb8837\" (UID: \"df181c99-e72f-4b8e-9a08-e416dbfb8837\") "
Nov 02 13:59:46.668274 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:46.668233    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df181c99-e72f-4b8e-9a08-e416dbfb8837-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df181c99-e72f-4b8e-9a08-e416dbfb8837-maas-api-xks-ca\") pod \"df181c99-e72f-4b8e-9a08-e416dbfb8837\" (UID: \"df181c99-e72f-4b8e-9a08-e416dbfb8837\") "
Nov 02 13:59:46.668519 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:46.668291    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df181c99-e72f-4b8e-9a08-e416dbfb8837-kube-api-access-7s88k\" (UniqueName: \"kubernetes.io/projected/df181c99-e72f-4b8e-9a08-e416dbfb8837-kube-api-access-7s88k\") pod \"df181c99-e72f-4b8e-9a08-e416dbfb8837\" (UID: \"df181c99-e72f-4b8e-9a08-e416dbfb8837\") "
Nov 02 13:59:46.668587 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:46.668551    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df181c99-e72f-4b8e-9a08-e416dbfb8837-maas-api-openshift-ca" pod "df181c99-e72f-4b8e-9a08-e416dbfb8837" (UID: "df181c99-e72f-4b8e-9a08-e416dbfb8837"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:46.668749 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:46.668717    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df181c99-e72f-4b8e-9a08-e416dbfb8837-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:59:46.670390 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:46.670361    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df181c99-e72f-4b8e-9a08-e416dbfb8837-maas-api-xks-ca" pod "df181c99-e72f-4b8e-9a08-e416dbfb8837" (UID: "df181c99-e72f-4b8e-9a08-e416dbfb8837"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:46.670489 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:46.670473    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df181c99-e72f-4b8e-9a08-e416dbfb8837-kube-api-access-7s88k" pod "df181c99-e72f-4b8e-9a08-e416dbfb8837" (UID: "df181c99-e72f-4b8e-9a08-e416dbfb8837"). InnerVolumeSpecName "kube-api-access-7s88k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:46.769513 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:46.769480    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df181c99-e72f-4b8e-9a08-e416dbfb8837-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:59:46.769513 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:46.769507    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s88k\" (UniqueName: \"kubernetes.io/projected/df181c99-e72f-4b8e-9a08-e416dbfb8837-kube-api-access-7s88k\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:59:47.361231 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:47.361193    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj" event={"ID":"df181c99-e72f-4b8e-9a08-e416dbfb8837","Type":"ContainerDied","Data":"960a50e7ba16015250ffd7315b2ad68eb02f9133e6442e2d1d1b9a5cb20f7375"}
Nov 02 13:59:47.361231 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:47.361212    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4acbc8-bvpmj"
Nov 02 13:59:47.361500 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:47.361241    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="960a50e7ba16015250ffd7315b2ad68eb02f9133e6442e2d1d1b9a5cb20f7375"
Nov 02 13:59:54.905259 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:54.905181    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw"]
Nov 02 13:59:54.905925 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:54.905901    4464 cpu_manager.go:414] "containerMap: removing container" podUID="df181c99-e72f-4b8e-9a08-e416dbfb8837" containerName="revoke-keys"
Nov 02 13:59:54.906040 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:54.905926    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df181c99-e72f-4b8e-9a08-e416dbfb8837" containerName="revoke-keys"
Nov 02 13:59:54.906116 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:54.906084    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="df181c99-e72f-4b8e-9a08-e416dbfb8837" containerName="revoke-keys"
Nov 02 13:59:54.909462 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:54.909440    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw"
Nov 02 13:59:54.944398 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:54.944367    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550a9c40-a0c5-4940-8680-83c6b02075c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8407b9d-2slkw\" (UID: \"550a9c40-a0c5-4940-8680-83c6b02075c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw"
Nov 02 13:59:54.944575 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:54.944426    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4wbx\" (UniqueName: \"kubernetes.io/projected/550a9c40-a0c5-4940-8680-83c6b02075c8-kube-api-access-v4wbx\") pod \"maas-api-revoke-sub-c8407b9d-2slkw\" (UID: \"550a9c40-a0c5-4940-8680-83c6b02075c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw"
Nov 02 13:59:54.944663 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:54.944581    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550a9c40-a0c5-4940-8680-83c6b02075c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8407b9d-2slkw\" (UID: \"550a9c40-a0c5-4940-8680-83c6b02075c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw"
Nov 02 13:59:54.965439 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:54.965411    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw"]
Nov 02 13:59:55.013129 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:55.013092    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:59:55.013290 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:55.013165    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:59:55.013290 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:55.013165    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:59:55.013290 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:55.013216    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 13:59:55.045683 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:55.045644    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550a9c40-a0c5-4940-8680-83c6b02075c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8407b9d-2slkw\" (UID: \"550a9c40-a0c5-4940-8680-83c6b02075c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw"
Nov 02 13:59:55.045830 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:55.045805    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4wbx\" (UniqueName: \"kubernetes.io/projected/550a9c40-a0c5-4940-8680-83c6b02075c8-kube-api-access-v4wbx\") pod \"maas-api-revoke-sub-c8407b9d-2slkw\" (UID: \"550a9c40-a0c5-4940-8680-83c6b02075c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw"
Nov 02 13:59:55.045915 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:55.045902    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550a9c40-a0c5-4940-8680-83c6b02075c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8407b9d-2slkw\" (UID: \"550a9c40-a0c5-4940-8680-83c6b02075c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw"
Nov 02 13:59:55.046323 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:55.046305    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550a9c40-a0c5-4940-8680-83c6b02075c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8407b9d-2slkw\" (UID: \"550a9c40-a0c5-4940-8680-83c6b02075c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw"
Nov 02 13:59:55.048188 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:55.048171    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550a9c40-a0c5-4940-8680-83c6b02075c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8407b9d-2slkw\" (UID: \"550a9c40-a0c5-4940-8680-83c6b02075c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw"
Nov 02 13:59:55.107555 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:55.107533    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4wbx\" (UniqueName: \"kubernetes.io/projected/550a9c40-a0c5-4940-8680-83c6b02075c8-kube-api-access-v4wbx\") pod \"maas-api-revoke-sub-c8407b9d-2slkw\" (UID: \"550a9c40-a0c5-4940-8680-83c6b02075c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw"
Nov 02 13:59:55.224899 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:55.224876    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw"
Nov 02 13:59:55.602963 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:55.602934    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw"]
Nov 02 13:59:55.604386 ip-10-0-133-239 kubenswrapper[4464]: W1102 13:59:55.604353    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550a9c40_a0c5_4940_8680_83c6b02075c8.slice/crio-652bd6cc5a6f78c72668bf4899e75f6e41432ae815baaa34f8980f4414f47c3b WatchSource:0}: Error finding container 652bd6cc5a6f78c72668bf4899e75f6e41432ae815baaa34f8980f4414f47c3b: Status 404 returned error can't find the container with id 652bd6cc5a6f78c72668bf4899e75f6e41432ae815baaa34f8980f4414f47c3b
Nov 02 13:59:56.416003 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:56.415955    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw" event={"ID":"550a9c40-a0c5-4940-8680-83c6b02075c8","Type":"ContainerStarted","Data":"d5df49ba5ce4182a4acbb48be3211da6c56a1ee942a36334fcc801898575c228"}
Nov 02 13:59:56.416411 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:56.416020    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw" event={"ID":"550a9c40-a0c5-4940-8680-83c6b02075c8","Type":"ContainerStarted","Data":"652bd6cc5a6f78c72668bf4899e75f6e41432ae815baaa34f8980f4414f47c3b"}
Nov 02 13:59:56.536080 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:56.536024    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw" podStartSLOduration=2.536008353 podStartE2EDuration="2.536008353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:56.534482447 +0000 UTC m=+7425.098190855" watchObservedRunningTime="2026-11-02 13:59:56.536008353 +0000 UTC m=+7425.099716762"
Nov 02 13:59:57.423445 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:57.423404    4464 generic.go:356] "Generic (PLEG): container finished" podID="550a9c40-a0c5-4940-8680-83c6b02075c8" containerID="d5df49ba5ce4182a4acbb48be3211da6c56a1ee942a36334fcc801898575c228" exitCode=0
Nov 02 13:59:57.423940 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:57.423487    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw" event={"ID":"550a9c40-a0c5-4940-8680-83c6b02075c8","Type":"ContainerDied","Data":"d5df49ba5ce4182a4acbb48be3211da6c56a1ee942a36334fcc801898575c228"}
Nov 02 13:59:58.290704 ip-10-0-133-239 kubenswrapper[4464]: E1102 13:59:58.290672    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 13:59:58.565731 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:58.565703    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw"
Nov 02 13:59:58.682000 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:58.681869    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/550a9c40-a0c5-4940-8680-83c6b02075c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550a9c40-a0c5-4940-8680-83c6b02075c8-maas-api-openshift-ca\") pod \"550a9c40-a0c5-4940-8680-83c6b02075c8\" (UID: \"550a9c40-a0c5-4940-8680-83c6b02075c8\") "
Nov 02 13:59:58.685884 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:58.685844    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/550a9c40-a0c5-4940-8680-83c6b02075c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550a9c40-a0c5-4940-8680-83c6b02075c8-maas-api-xks-ca\") pod \"550a9c40-a0c5-4940-8680-83c6b02075c8\" (UID: \"550a9c40-a0c5-4940-8680-83c6b02075c8\") "
Nov 02 13:59:58.686146 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:58.685957    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550a9c40-a0c5-4940-8680-83c6b02075c8-kube-api-access-v4wbx\" (UniqueName: \"kubernetes.io/projected/550a9c40-a0c5-4940-8680-83c6b02075c8-kube-api-access-v4wbx\") pod \"550a9c40-a0c5-4940-8680-83c6b02075c8\" (UID: \"550a9c40-a0c5-4940-8680-83c6b02075c8\") "
Nov 02 13:59:58.687029 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:58.687001    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550a9c40-a0c5-4940-8680-83c6b02075c8-maas-api-openshift-ca" pod "550a9c40-a0c5-4940-8680-83c6b02075c8" (UID: "550a9c40-a0c5-4940-8680-83c6b02075c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:58.706158 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:58.706097    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/550a9c40-a0c5-4940-8680-83c6b02075c8-maas-api-xks-ca" pod "550a9c40-a0c5-4940-8680-83c6b02075c8" (UID: "550a9c40-a0c5-4940-8680-83c6b02075c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:58.706704 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:58.706669    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550a9c40-a0c5-4940-8680-83c6b02075c8-kube-api-access-v4wbx" pod "550a9c40-a0c5-4940-8680-83c6b02075c8" (UID: "550a9c40-a0c5-4940-8680-83c6b02075c8"). InnerVolumeSpecName "kube-api-access-v4wbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:58.787830 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:58.787790    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4wbx\" (UniqueName: \"kubernetes.io/projected/550a9c40-a0c5-4940-8680-83c6b02075c8-kube-api-access-v4wbx\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:59:58.787830 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:58.787824    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550a9c40-a0c5-4940-8680-83c6b02075c8-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:59:58.788097 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:58.787840    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550a9c40-a0c5-4940-8680-83c6b02075c8-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 13:59:59.439146 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:59.439059    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw" event={"ID":"550a9c40-a0c5-4940-8680-83c6b02075c8","Type":"ContainerDied","Data":"652bd6cc5a6f78c72668bf4899e75f6e41432ae815baaa34f8980f4414f47c3b"}
Nov 02 13:59:59.439366 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:59.439155    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="652bd6cc5a6f78c72668bf4899e75f6e41432ae815baaa34f8980f4414f47c3b"
Nov 02 13:59:59.439366 ip-10-0-133-239 kubenswrapper[4464]: I1102 13:59:59.439076    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8407b9d-2slkw"
Nov 02 14:00:00.197851 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:00.194438    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv"]
Nov 02 14:00:00.197851 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:00.195181    4464 cpu_manager.go:414] "containerMap: removing container" podUID="550a9c40-a0c5-4940-8680-83c6b02075c8" containerName="revoke-keys"
Nov 02 14:00:00.197851 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:00.195203    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550a9c40-a0c5-4940-8680-83c6b02075c8" containerName="revoke-keys"
Nov 02 14:00:00.197851 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:00.195347    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="550a9c40-a0c5-4940-8680-83c6b02075c8" containerName="revoke-keys"
Nov 02 14:00:00.224999 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:00.224399    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv"
Nov 02 14:00:00.248999 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:00.244449    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv"]
Nov 02 14:00:00.319001 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:00.290247    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:00:00.319001 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:00.290947    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 14:00:00.319001 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:00.291049    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:00:00.372266 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:00.372227    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8nds\" (UniqueName: \"kubernetes.io/projected/60dcfcd8-3415-4432-8e41-807ffbbb7dfb-kube-api-access-c8nds\") pod \"maas-api-key-cleanup-29893800-psbfv\" (UID: \"60dcfcd8-3415-4432-8e41-807ffbbb7dfb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv"
Nov 02 14:00:00.373398 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:00.373370    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60dcfcd8-3415-4432-8e41-807ffbbb7dfb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893800-psbfv\" (UID: \"60dcfcd8-3415-4432-8e41-807ffbbb7dfb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv"
Nov 02 14:00:00.490057 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:00.486564    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8nds\" (UniqueName: \"kubernetes.io/projected/60dcfcd8-3415-4432-8e41-807ffbbb7dfb-kube-api-access-c8nds\") pod \"maas-api-key-cleanup-29893800-psbfv\" (UID: \"60dcfcd8-3415-4432-8e41-807ffbbb7dfb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv"
Nov 02 14:00:00.490057 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:00.486620    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60dcfcd8-3415-4432-8e41-807ffbbb7dfb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893800-psbfv\" (UID: \"60dcfcd8-3415-4432-8e41-807ffbbb7dfb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv"
Nov 02 14:00:00.505019 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:00.504962    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60dcfcd8-3415-4432-8e41-807ffbbb7dfb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893800-psbfv\" (UID: \"60dcfcd8-3415-4432-8e41-807ffbbb7dfb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv"
Nov 02 14:00:00.588003 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:00.584115    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8nds\" (UniqueName: \"kubernetes.io/projected/60dcfcd8-3415-4432-8e41-807ffbbb7dfb-kube-api-access-c8nds\") pod \"maas-api-key-cleanup-29893800-psbfv\" (UID: \"60dcfcd8-3415-4432-8e41-807ffbbb7dfb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv"
Nov 02 14:00:00.699186 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:00.699141    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv"
Nov 02 14:00:01.520169 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:01.520124    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv"]
Nov 02 14:00:01.562253 ip-10-0-133-239 kubenswrapper[4464]: W1102 14:00:01.562220    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60dcfcd8_3415_4432_8e41_807ffbbb7dfb.slice/crio-82c6dea597893d4af1cc28da2f3b9bcaf514917a1ad9e789082d871da3103324 WatchSource:0}: Error finding container 82c6dea597893d4af1cc28da2f3b9bcaf514917a1ad9e789082d871da3103324: Status 404 returned error can't find the container with id 82c6dea597893d4af1cc28da2f3b9bcaf514917a1ad9e789082d871da3103324
Nov 02 14:00:02.585080 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:02.585041    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv" event={"ID":"60dcfcd8-3415-4432-8e41-807ffbbb7dfb","Type":"ContainerStarted","Data":"61e9ddae1c9eaf24c90d0848f9e250f520d619591a84436b86ed36fa34ac3656"}
Nov 02 14:00:02.585668 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:02.585643    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv" event={"ID":"60dcfcd8-3415-4432-8e41-807ffbbb7dfb","Type":"ContainerStarted","Data":"82c6dea597893d4af1cc28da2f3b9bcaf514917a1ad9e789082d871da3103324"}
Nov 02 14:00:02.775010 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:02.774404    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv" podStartSLOduration=2.774382299 podStartE2EDuration="2.774382299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:02.769699023 +0000 UTC m=+7431.333407435" watchObservedRunningTime="2026-11-02 14:00:02.774382299 +0000 UTC m=+7431.338090709"
Nov 02 14:00:03.591405 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:03.591368    4464 generic.go:356] "Generic (PLEG): container finished" podID="60dcfcd8-3415-4432-8e41-807ffbbb7dfb" containerID="61e9ddae1c9eaf24c90d0848f9e250f520d619591a84436b86ed36fa34ac3656" exitCode=60
Nov 02 14:00:03.591846 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:03.591433    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv" event={"ID":"60dcfcd8-3415-4432-8e41-807ffbbb7dfb","Type":"ContainerDied","Data":"61e9ddae1c9eaf24c90d0848f9e250f520d619591a84436b86ed36fa34ac3656"}
Nov 02 14:00:03.591846 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:03.591663    4464 scope.go:122] "RemoveContainer" containerID="61e9ddae1c9eaf24c90d0848f9e250f520d619591a84436b86ed36fa34ac3656"
Nov 02 14:00:04.598083 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:04.598049    4464 generic.go:356] "Generic (PLEG): container finished" podID="60dcfcd8-3415-4432-8e41-807ffbbb7dfb" containerID="47aa76c21aaa4adf3fe51347249b98c833953a835f0e3e509ee4ea786d7affd9" exitCode=60
Nov 02 14:00:04.598564 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:04.598129    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv" event={"ID":"60dcfcd8-3415-4432-8e41-807ffbbb7dfb","Type":"ContainerDied","Data":"47aa76c21aaa4adf3fe51347249b98c833953a835f0e3e509ee4ea786d7affd9"}
Nov 02 14:00:04.598564 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:04.598186    4464 scope.go:122] "RemoveContainer" containerID="61e9ddae1c9eaf24c90d0848f9e250f520d619591a84436b86ed36fa34ac3656"
Nov 02 14:00:04.598564 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:04.598387    4464 scope.go:122] "RemoveContainer" containerID="47aa76c21aaa4adf3fe51347249b98c833953a835f0e3e509ee4ea786d7affd9"
Nov 02 14:00:04.598703 ip-10-0-133-239 kubenswrapper[4464]: E1102 14:00:04.598616    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893800-psbfv_odh-ai-gateway-infra(60dcfcd8-3415-4432-8e41-807ffbbb7dfb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv" podUID="60dcfcd8-3415-4432-8e41-807ffbbb7dfb"
Nov 02 14:00:05.597150 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:05.597116    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v"]
Nov 02 14:00:05.602050 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:05.602029    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v"
Nov 02 14:00:05.605110 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:05.605091    4464 scope.go:122] "RemoveContainer" containerID="47aa76c21aaa4adf3fe51347249b98c833953a835f0e3e509ee4ea786d7affd9"
Nov 02 14:00:05.605321 ip-10-0-133-239 kubenswrapper[4464]: E1102 14:00:05.605301    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893800-psbfv_odh-ai-gateway-infra(60dcfcd8-3415-4432-8e41-807ffbbb7dfb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv" podUID="60dcfcd8-3415-4432-8e41-807ffbbb7dfb"
Nov 02 14:00:05.655943 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:05.655785    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:05.708616 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:05.708587    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v"]
Nov 02 14:00:05.761091 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:05.761060    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwnpc\" (UniqueName: \"kubernetes.io/projected/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-kube-api-access-lwnpc\") pod \"maas-api-revoke-sub-3dedca5d-gjg9v\" (UID: \"1c4523a7-535b-42ba-a578-1b1b80d8b0e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v"
Nov 02 14:00:05.761236 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:05.761115    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dedca5d-gjg9v\" (UID: \"1c4523a7-535b-42ba-a578-1b1b80d8b0e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v"
Nov 02 14:00:05.761358 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:05.761338    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dedca5d-gjg9v\" (UID: \"1c4523a7-535b-42ba-a578-1b1b80d8b0e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v"
Nov 02 14:00:05.862060 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:05.861991    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dedca5d-gjg9v\" (UID: \"1c4523a7-535b-42ba-a578-1b1b80d8b0e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v"
Nov 02 14:00:05.862060 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:05.862042    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwnpc\" (UniqueName: \"kubernetes.io/projected/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-kube-api-access-lwnpc\") pod \"maas-api-revoke-sub-3dedca5d-gjg9v\" (UID: \"1c4523a7-535b-42ba-a578-1b1b80d8b0e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v"
Nov 02 14:00:05.862253 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:05.862092    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dedca5d-gjg9v\" (UID: \"1c4523a7-535b-42ba-a578-1b1b80d8b0e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v"
Nov 02 14:00:05.862759 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:05.862740    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dedca5d-gjg9v\" (UID: \"1c4523a7-535b-42ba-a578-1b1b80d8b0e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v"
Nov 02 14:00:05.864311 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:05.864289    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dedca5d-gjg9v\" (UID: \"1c4523a7-535b-42ba-a578-1b1b80d8b0e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v"
Nov 02 14:00:05.924196 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:05.924169    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwnpc\" (UniqueName: \"kubernetes.io/projected/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-kube-api-access-lwnpc\") pod \"maas-api-revoke-sub-3dedca5d-gjg9v\" (UID: \"1c4523a7-535b-42ba-a578-1b1b80d8b0e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v"
Nov 02 14:00:06.214803 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:06.214731    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v"
Nov 02 14:00:06.590626 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:06.590602    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v"]
Nov 02 14:00:06.590754 ip-10-0-133-239 kubenswrapper[4464]: W1102 14:00:06.590724    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4523a7_535b_42ba_a578_1b1b80d8b0e4.slice/crio-a6bcfede793540264f9d3667a828c84f049f30da6bf5a006755537ffa8784241 WatchSource:0}: Error finding container a6bcfede793540264f9d3667a828c84f049f30da6bf5a006755537ffa8784241: Status 404 returned error can't find the container with id a6bcfede793540264f9d3667a828c84f049f30da6bf5a006755537ffa8784241
Nov 02 14:00:06.611847 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:06.611815    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v" event={"ID":"1c4523a7-535b-42ba-a578-1b1b80d8b0e4","Type":"ContainerStarted","Data":"a6bcfede793540264f9d3667a828c84f049f30da6bf5a006755537ffa8784241"}
Nov 02 14:00:07.618153 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:07.618118    4464 generic.go:356] "Generic (PLEG): container finished" podID="1c4523a7-535b-42ba-a578-1b1b80d8b0e4" containerID="8a77b5445d4745eee7b4bcf3de6bcd4ab18fc1b930d231b8a7b2688d5fab9b52" exitCode=0
Nov 02 14:00:07.618622 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:07.618190    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v" event={"ID":"1c4523a7-535b-42ba-a578-1b1b80d8b0e4","Type":"ContainerDied","Data":"8a77b5445d4745eee7b4bcf3de6bcd4ab18fc1b930d231b8a7b2688d5fab9b52"}
Nov 02 14:00:08.757640 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:08.757614    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v"
Nov 02 14:00:08.890784 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:08.890711    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-maas-api-openshift-ca\") pod \"1c4523a7-535b-42ba-a578-1b1b80d8b0e4\" (UID: \"1c4523a7-535b-42ba-a578-1b1b80d8b0e4\") "
Nov 02 14:00:08.890929 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:08.890803    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-maas-api-xks-ca\") pod \"1c4523a7-535b-42ba-a578-1b1b80d8b0e4\" (UID: \"1c4523a7-535b-42ba-a578-1b1b80d8b0e4\") "
Nov 02 14:00:08.890929 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:08.890837    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-kube-api-access-lwnpc\" (UniqueName: \"kubernetes.io/projected/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-kube-api-access-lwnpc\") pod \"1c4523a7-535b-42ba-a578-1b1b80d8b0e4\" (UID: \"1c4523a7-535b-42ba-a578-1b1b80d8b0e4\") "
Nov 02 14:00:08.891167 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:08.891139    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-maas-api-openshift-ca" pod "1c4523a7-535b-42ba-a578-1b1b80d8b0e4" (UID: "1c4523a7-535b-42ba-a578-1b1b80d8b0e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:08.891286 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:08.891214    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:00:08.892803 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:08.892778    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-maas-api-xks-ca" pod "1c4523a7-535b-42ba-a578-1b1b80d8b0e4" (UID: "1c4523a7-535b-42ba-a578-1b1b80d8b0e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:08.892970 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:08.892943    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-kube-api-access-lwnpc" pod "1c4523a7-535b-42ba-a578-1b1b80d8b0e4" (UID: "1c4523a7-535b-42ba-a578-1b1b80d8b0e4"). InnerVolumeSpecName "kube-api-access-lwnpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:08.992468 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:08.992422    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:00:08.992468 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:08.992462    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwnpc\" (UniqueName: \"kubernetes.io/projected/1c4523a7-535b-42ba-a578-1b1b80d8b0e4-kube-api-access-lwnpc\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:00:09.630515 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:09.630484    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v" event={"ID":"1c4523a7-535b-42ba-a578-1b1b80d8b0e4","Type":"ContainerDied","Data":"a6bcfede793540264f9d3667a828c84f049f30da6bf5a006755537ffa8784241"}
Nov 02 14:00:09.630673 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:09.630522    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dedca5d-gjg9v"
Nov 02 14:00:09.630673 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:09.630530    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6bcfede793540264f9d3667a828c84f049f30da6bf5a006755537ffa8784241"
Nov 02 14:00:16.449342 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:16.449306    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4"]
Nov 02 14:00:16.449820 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:16.449751    4464 cpu_manager.go:414] "containerMap: removing container" podUID="1c4523a7-535b-42ba-a578-1b1b80d8b0e4" containerName="revoke-keys"
Nov 02 14:00:16.449820 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:16.449762    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4523a7-535b-42ba-a578-1b1b80d8b0e4" containerName="revoke-keys"
Nov 02 14:00:16.450011 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:16.449985    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4523a7-535b-42ba-a578-1b1b80d8b0e4" containerName="revoke-keys"
Nov 02 14:00:16.453438 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:16.453415    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4"
Nov 02 14:00:16.512854 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:16.512823    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4"]
Nov 02 14:00:16.559155 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:16.559126    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:16.559470 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:16.559449    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96512c88-7b5c-442f-a798-9a0bd0995c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70600120-l99v4\" (UID: \"96512c88-7b5c-442f-a798-9a0bd0995c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4"
Nov 02 14:00:16.559624 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:16.559602    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-777ck\" (UniqueName: \"kubernetes.io/projected/96512c88-7b5c-442f-a798-9a0bd0995c1c-kube-api-access-777ck\") pod \"maas-api-revoke-sub-70600120-l99v4\" (UID: \"96512c88-7b5c-442f-a798-9a0bd0995c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4"
Nov 02 14:00:16.559687 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:16.559645    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96512c88-7b5c-442f-a798-9a0bd0995c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70600120-l99v4\" (UID: \"96512c88-7b5c-442f-a798-9a0bd0995c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4"
Nov 02 14:00:16.660361 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:16.660324    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96512c88-7b5c-442f-a798-9a0bd0995c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70600120-l99v4\" (UID: \"96512c88-7b5c-442f-a798-9a0bd0995c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4"
Nov 02 14:00:16.660575 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:16.660440    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-777ck\" (UniqueName: \"kubernetes.io/projected/96512c88-7b5c-442f-a798-9a0bd0995c1c-kube-api-access-777ck\") pod \"maas-api-revoke-sub-70600120-l99v4\" (UID: \"96512c88-7b5c-442f-a798-9a0bd0995c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4"
Nov 02 14:00:16.660575 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:16.660473    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96512c88-7b5c-442f-a798-9a0bd0995c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70600120-l99v4\" (UID: \"96512c88-7b5c-442f-a798-9a0bd0995c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4"
Nov 02 14:00:16.661028 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:16.661001    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96512c88-7b5c-442f-a798-9a0bd0995c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70600120-l99v4\" (UID: \"96512c88-7b5c-442f-a798-9a0bd0995c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4"
Nov 02 14:00:16.662991 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:16.662939    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96512c88-7b5c-442f-a798-9a0bd0995c1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70600120-l99v4\" (UID: \"96512c88-7b5c-442f-a798-9a0bd0995c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4"
Nov 02 14:00:16.723191 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:16.723156    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-777ck\" (UniqueName: \"kubernetes.io/projected/96512c88-7b5c-442f-a798-9a0bd0995c1c-kube-api-access-777ck\") pod \"maas-api-revoke-sub-70600120-l99v4\" (UID: \"96512c88-7b5c-442f-a798-9a0bd0995c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4"
Nov 02 14:00:16.766004 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:16.765955    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4"
Nov 02 14:00:17.138374 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:17.138344    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4"]
Nov 02 14:00:17.139356 ip-10-0-133-239 kubenswrapper[4464]: W1102 14:00:17.139330    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96512c88_7b5c_442f_a798_9a0bd0995c1c.slice/crio-5d9ead319244577acee3d3a7c2a8132da9b67c766d4a29afcb2b76c03887e2d7 WatchSource:0}: Error finding container 5d9ead319244577acee3d3a7c2a8132da9b67c766d4a29afcb2b76c03887e2d7: Status 404 returned error can't find the container with id 5d9ead319244577acee3d3a7c2a8132da9b67c766d4a29afcb2b76c03887e2d7
Nov 02 14:00:17.575553 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:17.575520    4464 scope.go:122] "RemoveContainer" containerID="47aa76c21aaa4adf3fe51347249b98c833953a835f0e3e509ee4ea786d7affd9"
Nov 02 14:00:17.674165 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:17.674126    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4" event={"ID":"96512c88-7b5c-442f-a798-9a0bd0995c1c","Type":"ContainerStarted","Data":"442d1a5b5db37c72c856fff137b015bb8d8560b42367bfbc4e12513a3463c590"}
Nov 02 14:00:17.674319 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:17.674185    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4" event={"ID":"96512c88-7b5c-442f-a798-9a0bd0995c1c","Type":"ContainerStarted","Data":"5d9ead319244577acee3d3a7c2a8132da9b67c766d4a29afcb2b76c03887e2d7"}
Nov 02 14:00:17.794274 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:17.794223    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4" podStartSLOduration=1.794209406 podStartE2EDuration="1.794209406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:17.792703546 +0000 UTC m=+7446.356411967" watchObservedRunningTime="2026-11-02 14:00:17.794209406 +0000 UTC m=+7446.357917814"
Nov 02 14:00:18.681376 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:18.681291    4464 generic.go:356] "Generic (PLEG): container finished" podID="60dcfcd8-3415-4432-8e41-807ffbbb7dfb" containerID="39ba078243e0ecb885ad930dba757e45a197c40eefc13ec23e1fc7de30e8a69f" exitCode=60
Nov 02 14:00:18.681376 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:18.681360    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv" event={"ID":"60dcfcd8-3415-4432-8e41-807ffbbb7dfb","Type":"ContainerDied","Data":"39ba078243e0ecb885ad930dba757e45a197c40eefc13ec23e1fc7de30e8a69f"}
Nov 02 14:00:18.681890 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:18.681415    4464 scope.go:122] "RemoveContainer" containerID="47aa76c21aaa4adf3fe51347249b98c833953a835f0e3e509ee4ea786d7affd9"
Nov 02 14:00:18.681890 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:18.681711    4464 scope.go:122] "RemoveContainer" containerID="39ba078243e0ecb885ad930dba757e45a197c40eefc13ec23e1fc7de30e8a69f"
Nov 02 14:00:18.681997 ip-10-0-133-239 kubenswrapper[4464]: E1102 14:00:18.681935    4464 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893800-psbfv_odh-ai-gateway-infra(60dcfcd8-3415-4432-8e41-807ffbbb7dfb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv" podUID="60dcfcd8-3415-4432-8e41-807ffbbb7dfb"
Nov 02 14:00:18.683005 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:18.682963    4464 generic.go:356] "Generic (PLEG): container finished" podID="96512c88-7b5c-442f-a798-9a0bd0995c1c" containerID="442d1a5b5db37c72c856fff137b015bb8d8560b42367bfbc4e12513a3463c590" exitCode=0
Nov 02 14:00:18.683106 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:18.683000    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4" event={"ID":"96512c88-7b5c-442f-a798-9a0bd0995c1c","Type":"ContainerDied","Data":"442d1a5b5db37c72c856fff137b015bb8d8560b42367bfbc4e12513a3463c590"}
Nov 02 14:00:19.813138 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:19.813093    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv"]
Nov 02 14:00:19.895332 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:19.895300    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4"
Nov 02 14:00:19.995769 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:19.995725    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96512c88-7b5c-442f-a798-9a0bd0995c1c-kube-api-access-777ck\" (UniqueName: \"kubernetes.io/projected/96512c88-7b5c-442f-a798-9a0bd0995c1c-kube-api-access-777ck\") pod \"96512c88-7b5c-442f-a798-9a0bd0995c1c\" (UID: \"96512c88-7b5c-442f-a798-9a0bd0995c1c\") "
Nov 02 14:00:19.995950 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:19.995825    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96512c88-7b5c-442f-a798-9a0bd0995c1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96512c88-7b5c-442f-a798-9a0bd0995c1c-maas-api-xks-ca\") pod \"96512c88-7b5c-442f-a798-9a0bd0995c1c\" (UID: \"96512c88-7b5c-442f-a798-9a0bd0995c1c\") "
Nov 02 14:00:19.995950 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:19.995899    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96512c88-7b5c-442f-a798-9a0bd0995c1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96512c88-7b5c-442f-a798-9a0bd0995c1c-maas-api-openshift-ca\") pod \"96512c88-7b5c-442f-a798-9a0bd0995c1c\" (UID: \"96512c88-7b5c-442f-a798-9a0bd0995c1c\") "
Nov 02 14:00:19.996629 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:19.996588    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96512c88-7b5c-442f-a798-9a0bd0995c1c-maas-api-openshift-ca" pod "96512c88-7b5c-442f-a798-9a0bd0995c1c" (UID: "96512c88-7b5c-442f-a798-9a0bd0995c1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:19.997540 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:19.997502    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96512c88-7b5c-442f-a798-9a0bd0995c1c-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:00:19.998684 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:19.998642    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96512c88-7b5c-442f-a798-9a0bd0995c1c-kube-api-access-777ck" pod "96512c88-7b5c-442f-a798-9a0bd0995c1c" (UID: "96512c88-7b5c-442f-a798-9a0bd0995c1c"). InnerVolumeSpecName "kube-api-access-777ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:19.999653 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:19.999627    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96512c88-7b5c-442f-a798-9a0bd0995c1c-maas-api-xks-ca" pod "96512c88-7b5c-442f-a798-9a0bd0995c1c" (UID: "96512c88-7b5c-442f-a798-9a0bd0995c1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:20.024955 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:20.024911    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv"
Nov 02 14:00:20.098909 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:20.098860    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60dcfcd8-3415-4432-8e41-807ffbbb7dfb-kube-api-access-c8nds\" (UniqueName: \"kubernetes.io/projected/60dcfcd8-3415-4432-8e41-807ffbbb7dfb-kube-api-access-c8nds\") pod \"60dcfcd8-3415-4432-8e41-807ffbbb7dfb\" (UID: \"60dcfcd8-3415-4432-8e41-807ffbbb7dfb\") "
Nov 02 14:00:20.099141 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:20.098916    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60dcfcd8-3415-4432-8e41-807ffbbb7dfb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60dcfcd8-3415-4432-8e41-807ffbbb7dfb-maas-api-service-ca\") pod \"60dcfcd8-3415-4432-8e41-807ffbbb7dfb\" (UID: \"60dcfcd8-3415-4432-8e41-807ffbbb7dfb\") "
Nov 02 14:00:20.099393 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:20.099366    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-777ck\" (UniqueName: \"kubernetes.io/projected/96512c88-7b5c-442f-a798-9a0bd0995c1c-kube-api-access-777ck\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.099495 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:20.099400    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96512c88-7b5c-442f-a798-9a0bd0995c1c-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.099909 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:20.099882    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60dcfcd8-3415-4432-8e41-807ffbbb7dfb-maas-api-service-ca" pod "60dcfcd8-3415-4432-8e41-807ffbbb7dfb" (UID: "60dcfcd8-3415-4432-8e41-807ffbbb7dfb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:20.101721 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:20.101693    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60dcfcd8-3415-4432-8e41-807ffbbb7dfb-kube-api-access-c8nds" pod "60dcfcd8-3415-4432-8e41-807ffbbb7dfb" (UID: "60dcfcd8-3415-4432-8e41-807ffbbb7dfb"). InnerVolumeSpecName "kube-api-access-c8nds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:20.207027 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:20.200673    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8nds\" (UniqueName: \"kubernetes.io/projected/60dcfcd8-3415-4432-8e41-807ffbbb7dfb-kube-api-access-c8nds\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.207027 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:20.200713    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60dcfcd8-3415-4432-8e41-807ffbbb7dfb-maas-api-service-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:00:20.716629 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:20.716590    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4" event={"ID":"96512c88-7b5c-442f-a798-9a0bd0995c1c","Type":"ContainerDied","Data":"5d9ead319244577acee3d3a7c2a8132da9b67c766d4a29afcb2b76c03887e2d7"}
Nov 02 14:00:20.717024 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:20.717001    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d9ead319244577acee3d3a7c2a8132da9b67c766d4a29afcb2b76c03887e2d7"
Nov 02 14:00:20.717185 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:20.716916    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70600120-l99v4"
Nov 02 14:00:20.726998 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:20.723090    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv" event={"ID":"60dcfcd8-3415-4432-8e41-807ffbbb7dfb","Type":"ContainerDied","Data":"82c6dea597893d4af1cc28da2f3b9bcaf514917a1ad9e789082d871da3103324"}
Nov 02 14:00:20.727528 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:20.727502    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv"
Nov 02 14:00:20.739468 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:20.727670    4464 scope.go:122] "RemoveContainer" containerID="39ba078243e0ecb885ad930dba757e45a197c40eefc13ec23e1fc7de30e8a69f"
Nov 02 14:00:20.925007 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:20.921748    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv"]
Nov 02 14:00:20.936998 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:20.935178    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-psbfv"]
Nov 02 14:00:21.618000 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:21.610861    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60dcfcd8-3415-4432-8e41-807ffbbb7dfb" path="/var/lib/kubelet/pods/60dcfcd8-3415-4432-8e41-807ffbbb7dfb/volumes"
Nov 02 14:00:27.264566 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.264521    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99"]
Nov 02 14:00:27.265211 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.265190    4464 cpu_manager.go:414] "containerMap: removing container" podUID="60dcfcd8-3415-4432-8e41-807ffbbb7dfb" containerName="cleanup"
Nov 02 14:00:27.265313 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.265213    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60dcfcd8-3415-4432-8e41-807ffbbb7dfb" containerName="cleanup"
Nov 02 14:00:27.265313 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.265253    4464 cpu_manager.go:414] "containerMap: removing container" podUID="96512c88-7b5c-442f-a798-9a0bd0995c1c" containerName="revoke-keys"
Nov 02 14:00:27.265313 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.265262    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96512c88-7b5c-442f-a798-9a0bd0995c1c" containerName="revoke-keys"
Nov 02 14:00:27.265313 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.265280    4464 cpu_manager.go:414] "containerMap: removing container" podUID="60dcfcd8-3415-4432-8e41-807ffbbb7dfb" containerName="cleanup"
Nov 02 14:00:27.265313 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.265288    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60dcfcd8-3415-4432-8e41-807ffbbb7dfb" containerName="cleanup"
Nov 02 14:00:27.265313 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.265301    4464 cpu_manager.go:414] "containerMap: removing container" podUID="60dcfcd8-3415-4432-8e41-807ffbbb7dfb" containerName="cleanup"
Nov 02 14:00:27.265313 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.265309    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60dcfcd8-3415-4432-8e41-807ffbbb7dfb" containerName="cleanup"
Nov 02 14:00:27.265684 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.265421    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="96512c88-7b5c-442f-a798-9a0bd0995c1c" containerName="revoke-keys"
Nov 02 14:00:27.265684 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.265437    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="60dcfcd8-3415-4432-8e41-807ffbbb7dfb" containerName="cleanup"
Nov 02 14:00:27.265684 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.265447    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="60dcfcd8-3415-4432-8e41-807ffbbb7dfb" containerName="cleanup"
Nov 02 14:00:27.270136 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.270116    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99"
Nov 02 14:00:27.324207 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.324175    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d89b7fa8-t2x99\" (UID: \"7adbd0a3-bbe9-4922-81c0-af50e8432e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99"
Nov 02 14:00:27.324510 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.324485    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d89b7fa8-t2x99\" (UID: \"7adbd0a3-bbe9-4922-81c0-af50e8432e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99"
Nov 02 14:00:27.324694 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.324672    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz8r2\" (UniqueName: \"kubernetes.io/projected/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-kube-api-access-nz8r2\") pod \"maas-api-revoke-sub-d89b7fa8-t2x99\" (UID: \"7adbd0a3-bbe9-4922-81c0-af50e8432e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99"
Nov 02 14:00:27.327594 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.327567    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99"]
Nov 02 14:00:27.374481 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.374453    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:27.374481 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.374466    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 14:00:27.374481 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.374465    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:00:27.374481 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.374485    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:00:27.425425 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.425397    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d89b7fa8-t2x99\" (UID: \"7adbd0a3-bbe9-4922-81c0-af50e8432e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99"
Nov 02 14:00:27.425624 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.425459    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d89b7fa8-t2x99\" (UID: \"7adbd0a3-bbe9-4922-81c0-af50e8432e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99"
Nov 02 14:00:27.425624 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.425501    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz8r2\" (UniqueName: \"kubernetes.io/projected/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-kube-api-access-nz8r2\") pod \"maas-api-revoke-sub-d89b7fa8-t2x99\" (UID: \"7adbd0a3-bbe9-4922-81c0-af50e8432e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99"
Nov 02 14:00:27.426234 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.426210    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d89b7fa8-t2x99\" (UID: \"7adbd0a3-bbe9-4922-81c0-af50e8432e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99"
Nov 02 14:00:27.427865 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.427833    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d89b7fa8-t2x99\" (UID: \"7adbd0a3-bbe9-4922-81c0-af50e8432e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99"
Nov 02 14:00:27.490687 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.490653    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz8r2\" (UniqueName: \"kubernetes.io/projected/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-kube-api-access-nz8r2\") pod \"maas-api-revoke-sub-d89b7fa8-t2x99\" (UID: \"7adbd0a3-bbe9-4922-81c0-af50e8432e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99"
Nov 02 14:00:27.582451 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.582382    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99"
Nov 02 14:00:27.957714 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:27.957679    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99"]
Nov 02 14:00:27.958779 ip-10-0-133-239 kubenswrapper[4464]: W1102 14:00:27.958748    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7adbd0a3_bbe9_4922_81c0_af50e8432e0b.slice/crio-c06e9917bfd10dfc43333245b9f3d4f5bb7f89c0dfc36b49f242728739375356 WatchSource:0}: Error finding container c06e9917bfd10dfc43333245b9f3d4f5bb7f89c0dfc36b49f242728739375356: Status 404 returned error can't find the container with id c06e9917bfd10dfc43333245b9f3d4f5bb7f89c0dfc36b49f242728739375356
Nov 02 14:00:28.291308 ip-10-0-133-239 kubenswrapper[4464]: E1102 14:00:28.291269    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 14:00:28.785202 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:28.785164    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99" event={"ID":"7adbd0a3-bbe9-4922-81c0-af50e8432e0b","Type":"ContainerStarted","Data":"cc81ace925a3bd56c02394623c266a34b3252877fbc579144da88c645c06a834"}
Nov 02 14:00:28.785386 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:28.785217    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99" event={"ID":"7adbd0a3-bbe9-4922-81c0-af50e8432e0b","Type":"ContainerStarted","Data":"c06e9917bfd10dfc43333245b9f3d4f5bb7f89c0dfc36b49f242728739375356"}
Nov 02 14:00:28.907125 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:28.907058    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99" podStartSLOduration=1.9070401609999998 podStartE2EDuration="1.907040161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:28.903571806 +0000 UTC m=+7457.467280219" watchObservedRunningTime="2026-11-02 14:00:28.907040161 +0000 UTC m=+7457.470748570"
Nov 02 14:00:29.792164 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:29.792124    4464 generic.go:356] "Generic (PLEG): container finished" podID="7adbd0a3-bbe9-4922-81c0-af50e8432e0b" containerID="cc81ace925a3bd56c02394623c266a34b3252877fbc579144da88c645c06a834" exitCode=0
Nov 02 14:00:29.792556 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:29.792176    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99" event={"ID":"7adbd0a3-bbe9-4922-81c0-af50e8432e0b","Type":"ContainerDied","Data":"cc81ace925a3bd56c02394623c266a34b3252877fbc579144da88c645c06a834"}
Nov 02 14:00:30.934548 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:30.934524    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99"
Nov 02 14:00:30.962802 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:30.962769    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-maas-api-openshift-ca\") pod \"7adbd0a3-bbe9-4922-81c0-af50e8432e0b\" (UID: \"7adbd0a3-bbe9-4922-81c0-af50e8432e0b\") "
Nov 02 14:00:30.962959 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:30.962871    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-maas-api-xks-ca\") pod \"7adbd0a3-bbe9-4922-81c0-af50e8432e0b\" (UID: \"7adbd0a3-bbe9-4922-81c0-af50e8432e0b\") "
Nov 02 14:00:30.962959 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:30.962890    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-kube-api-access-nz8r2\" (UniqueName: \"kubernetes.io/projected/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-kube-api-access-nz8r2\") pod \"7adbd0a3-bbe9-4922-81c0-af50e8432e0b\" (UID: \"7adbd0a3-bbe9-4922-81c0-af50e8432e0b\") "
Nov 02 14:00:30.963144 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:30.963122    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-maas-api-openshift-ca" pod "7adbd0a3-bbe9-4922-81c0-af50e8432e0b" (UID: "7adbd0a3-bbe9-4922-81c0-af50e8432e0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:30.964941 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:30.964917    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-maas-api-xks-ca" pod "7adbd0a3-bbe9-4922-81c0-af50e8432e0b" (UID: "7adbd0a3-bbe9-4922-81c0-af50e8432e0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:30.965072 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:30.964963    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-kube-api-access-nz8r2" pod "7adbd0a3-bbe9-4922-81c0-af50e8432e0b" (UID: "7adbd0a3-bbe9-4922-81c0-af50e8432e0b"). InnerVolumeSpecName "kube-api-access-nz8r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:31.064664 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:31.064582    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:00:31.064664 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:31.064622    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:00:31.064664 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:31.064638    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz8r2\" (UniqueName: \"kubernetes.io/projected/7adbd0a3-bbe9-4922-81c0-af50e8432e0b-kube-api-access-nz8r2\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:00:31.808141 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:31.808100    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99" event={"ID":"7adbd0a3-bbe9-4922-81c0-af50e8432e0b","Type":"ContainerDied","Data":"c06e9917bfd10dfc43333245b9f3d4f5bb7f89c0dfc36b49f242728739375356"}
Nov 02 14:00:31.808376 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:31.808171    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c06e9917bfd10dfc43333245b9f3d4f5bb7f89c0dfc36b49f242728739375356"
Nov 02 14:00:31.808376 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:31.808109    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d89b7fa8-t2x99"
Nov 02 14:00:34.222156 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:34.222105    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-pk25x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:34.222850 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:34.222814    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-pk25x" podUID="6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:34.886200 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:34.886150    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dghm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:34.886508 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:34.886465    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7" podUID="2d10b658-29b5-4e07-84df-13326972bead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:35.038239 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:35.038198    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cnggl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:35.038578 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:35.038531    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl" podUID="91491db1-c3aa-422e-9e3d-5575b4197aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:37.963866 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:37.963814    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln"]
Nov 02 14:00:37.964401 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:37.964385    4464 cpu_manager.go:414] "containerMap: removing container" podUID="7adbd0a3-bbe9-4922-81c0-af50e8432e0b" containerName="revoke-keys"
Nov 02 14:00:37.964457 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:37.964402    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7adbd0a3-bbe9-4922-81c0-af50e8432e0b" containerName="revoke-keys"
Nov 02 14:00:37.964500 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:37.964487    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="60dcfcd8-3415-4432-8e41-807ffbbb7dfb" containerName="cleanup"
Nov 02 14:00:37.964500 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:37.964496    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="7adbd0a3-bbe9-4922-81c0-af50e8432e0b" containerName="revoke-keys"
Nov 02 14:00:37.967852 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:37.967837    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln"
Nov 02 14:00:38.073388 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:38.073348    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:00:38.073388 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:38.073386    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:38.073595 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:38.073401    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 14:00:38.073595 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:38.073404    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:00:38.076040 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:38.076019    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln"]
Nov 02 14:00:38.080909 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:38.080892    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da77def7-0134-478f-90bc-6e08b479798f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5008c1e7-zn2ln\" (UID: \"da77def7-0134-478f-90bc-6e08b479798f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln"
Nov 02 14:00:38.080993 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:38.080924    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4p5x\" (UniqueName: \"kubernetes.io/projected/da77def7-0134-478f-90bc-6e08b479798f-kube-api-access-l4p5x\") pod \"maas-api-revoke-sub-5008c1e7-zn2ln\" (UID: \"da77def7-0134-478f-90bc-6e08b479798f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln"
Nov 02 14:00:38.081164 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:38.081142    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da77def7-0134-478f-90bc-6e08b479798f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5008c1e7-zn2ln\" (UID: \"da77def7-0134-478f-90bc-6e08b479798f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln"
Nov 02 14:00:38.181713 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:38.181682    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da77def7-0134-478f-90bc-6e08b479798f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5008c1e7-zn2ln\" (UID: \"da77def7-0134-478f-90bc-6e08b479798f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln"
Nov 02 14:00:38.181713 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:38.181717    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4p5x\" (UniqueName: \"kubernetes.io/projected/da77def7-0134-478f-90bc-6e08b479798f-kube-api-access-l4p5x\") pod \"maas-api-revoke-sub-5008c1e7-zn2ln\" (UID: \"da77def7-0134-478f-90bc-6e08b479798f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln"
Nov 02 14:00:38.181928 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:38.181801    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da77def7-0134-478f-90bc-6e08b479798f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5008c1e7-zn2ln\" (UID: \"da77def7-0134-478f-90bc-6e08b479798f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln"
Nov 02 14:00:38.182414 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:38.182396    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da77def7-0134-478f-90bc-6e08b479798f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5008c1e7-zn2ln\" (UID: \"da77def7-0134-478f-90bc-6e08b479798f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln"
Nov 02 14:00:38.184024 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:38.184005    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da77def7-0134-478f-90bc-6e08b479798f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5008c1e7-zn2ln\" (UID: \"da77def7-0134-478f-90bc-6e08b479798f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln"
Nov 02 14:00:38.244076 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:38.244053    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4p5x\" (UniqueName: \"kubernetes.io/projected/da77def7-0134-478f-90bc-6e08b479798f-kube-api-access-l4p5x\") pod \"maas-api-revoke-sub-5008c1e7-zn2ln\" (UID: \"da77def7-0134-478f-90bc-6e08b479798f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln"
Nov 02 14:00:38.280698 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:38.280670    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln"
Nov 02 14:00:38.657334 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:38.657302    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln"]
Nov 02 14:00:38.657751 ip-10-0-133-239 kubenswrapper[4464]: W1102 14:00:38.657722    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda77def7_0134_478f_90bc_6e08b479798f.slice/crio-6ad6f5e54e1f4f30d32f16a32742dc3b0c2e440aecded4bd49e929ae16017065 WatchSource:0}: Error finding container 6ad6f5e54e1f4f30d32f16a32742dc3b0c2e440aecded4bd49e929ae16017065: Status 404 returned error can't find the container with id 6ad6f5e54e1f4f30d32f16a32742dc3b0c2e440aecded4bd49e929ae16017065
Nov 02 14:00:38.873528 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:38.873491    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln" event={"ID":"da77def7-0134-478f-90bc-6e08b479798f","Type":"ContainerStarted","Data":"6ad6f5e54e1f4f30d32f16a32742dc3b0c2e440aecded4bd49e929ae16017065"}
Nov 02 14:00:39.880576 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:39.880541    4464 generic.go:356] "Generic (PLEG): container finished" podID="da77def7-0134-478f-90bc-6e08b479798f" containerID="f21f43177e08f65be12c5fc99c9b206dd0d1c58dc285618b91fc989dbad9360c" exitCode=0
Nov 02 14:00:39.880944 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:39.880599    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln" event={"ID":"da77def7-0134-478f-90bc-6e08b479798f","Type":"ContainerDied","Data":"f21f43177e08f65be12c5fc99c9b206dd0d1c58dc285618b91fc989dbad9360c"}
Nov 02 14:00:41.022209 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:41.022182    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln"
Nov 02 14:00:41.110326 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:41.110292    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da77def7-0134-478f-90bc-6e08b479798f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da77def7-0134-478f-90bc-6e08b479798f-maas-api-openshift-ca\") pod \"da77def7-0134-478f-90bc-6e08b479798f\" (UID: \"da77def7-0134-478f-90bc-6e08b479798f\") "
Nov 02 14:00:41.110498 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:41.110415    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da77def7-0134-478f-90bc-6e08b479798f-kube-api-access-l4p5x\" (UniqueName: \"kubernetes.io/projected/da77def7-0134-478f-90bc-6e08b479798f-kube-api-access-l4p5x\") pod \"da77def7-0134-478f-90bc-6e08b479798f\" (UID: \"da77def7-0134-478f-90bc-6e08b479798f\") "
Nov 02 14:00:41.110577 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:41.110515    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da77def7-0134-478f-90bc-6e08b479798f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da77def7-0134-478f-90bc-6e08b479798f-maas-api-xks-ca\") pod \"da77def7-0134-478f-90bc-6e08b479798f\" (UID: \"da77def7-0134-478f-90bc-6e08b479798f\") "
Nov 02 14:00:41.110683 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:41.110656    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da77def7-0134-478f-90bc-6e08b479798f-maas-api-openshift-ca" pod "da77def7-0134-478f-90bc-6e08b479798f" (UID: "da77def7-0134-478f-90bc-6e08b479798f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:41.110817 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:41.110800    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da77def7-0134-478f-90bc-6e08b479798f-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:00:41.112458 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:41.112437    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da77def7-0134-478f-90bc-6e08b479798f-maas-api-xks-ca" pod "da77def7-0134-478f-90bc-6e08b479798f" (UID: "da77def7-0134-478f-90bc-6e08b479798f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:41.112566 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:41.112545    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da77def7-0134-478f-90bc-6e08b479798f-kube-api-access-l4p5x" pod "da77def7-0134-478f-90bc-6e08b479798f" (UID: "da77def7-0134-478f-90bc-6e08b479798f"). InnerVolumeSpecName "kube-api-access-l4p5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:41.212255 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:41.212179    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4p5x\" (UniqueName: \"kubernetes.io/projected/da77def7-0134-478f-90bc-6e08b479798f-kube-api-access-l4p5x\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:00:41.212255 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:41.212207    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da77def7-0134-478f-90bc-6e08b479798f-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:00:41.892311 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:41.892229    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln"
Nov 02 14:00:41.892311 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:41.892265    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5008c1e7-zn2ln" event={"ID":"da77def7-0134-478f-90bc-6e08b479798f","Type":"ContainerDied","Data":"6ad6f5e54e1f4f30d32f16a32742dc3b0c2e440aecded4bd49e929ae16017065"}
Nov 02 14:00:41.892503 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:41.892345    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ad6f5e54e1f4f30d32f16a32742dc3b0c2e440aecded4bd49e929ae16017065"
Nov 02 14:00:48.754923 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.754882    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv"]
Nov 02 14:00:48.755653 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.755408    4464 cpu_manager.go:414] "containerMap: removing container" podUID="da77def7-0134-478f-90bc-6e08b479798f" containerName="revoke-keys"
Nov 02 14:00:48.755653 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.755424    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da77def7-0134-478f-90bc-6e08b479798f" containerName="revoke-keys"
Nov 02 14:00:48.755653 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.755488    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="da77def7-0134-478f-90bc-6e08b479798f" containerName="revoke-keys"
Nov 02 14:00:48.758300 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.758278    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv"
Nov 02 14:00:48.815685 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.815655    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:00:48.816269 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.816111    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:48.866493 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.866456    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:00:48.866678 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.866556    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 14:00:48.869231 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.869211    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv"]
Nov 02 14:00:48.888121 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.888095    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139d0139-2193-482c-849f-e098d9811f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8b17e66-5kpkv\" (UID: \"139d0139-2193-482c-849f-e098d9811f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv"
Nov 02 14:00:48.888254 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.888151    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m9rq\" (UniqueName: \"kubernetes.io/projected/139d0139-2193-482c-849f-e098d9811f9f-kube-api-access-8m9rq\") pod \"maas-api-revoke-sub-e8b17e66-5kpkv\" (UID: \"139d0139-2193-482c-849f-e098d9811f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv"
Nov 02 14:00:48.888323 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.888299    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139d0139-2193-482c-849f-e098d9811f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8b17e66-5kpkv\" (UID: \"139d0139-2193-482c-849f-e098d9811f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv"
Nov 02 14:00:48.989451 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.989414    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139d0139-2193-482c-849f-e098d9811f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8b17e66-5kpkv\" (UID: \"139d0139-2193-482c-849f-e098d9811f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv"
Nov 02 14:00:48.989654 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.989537    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139d0139-2193-482c-849f-e098d9811f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8b17e66-5kpkv\" (UID: \"139d0139-2193-482c-849f-e098d9811f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv"
Nov 02 14:00:48.989654 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.989588    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m9rq\" (UniqueName: \"kubernetes.io/projected/139d0139-2193-482c-849f-e098d9811f9f-kube-api-access-8m9rq\") pod \"maas-api-revoke-sub-e8b17e66-5kpkv\" (UID: \"139d0139-2193-482c-849f-e098d9811f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv"
Nov 02 14:00:48.990163 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.990127    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139d0139-2193-482c-849f-e098d9811f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8b17e66-5kpkv\" (UID: \"139d0139-2193-482c-849f-e098d9811f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv"
Nov 02 14:00:48.991882 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:48.991856    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139d0139-2193-482c-849f-e098d9811f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8b17e66-5kpkv\" (UID: \"139d0139-2193-482c-849f-e098d9811f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv"
Nov 02 14:00:49.051746 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:49.051687    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m9rq\" (UniqueName: \"kubernetes.io/projected/139d0139-2193-482c-849f-e098d9811f9f-kube-api-access-8m9rq\") pod \"maas-api-revoke-sub-e8b17e66-5kpkv\" (UID: \"139d0139-2193-482c-849f-e098d9811f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv"
Nov 02 14:00:49.071266 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:49.071241    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv"
Nov 02 14:00:49.444551 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:49.444520    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv"]
Nov 02 14:00:49.444961 ip-10-0-133-239 kubenswrapper[4464]: W1102 14:00:49.444936    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139d0139_2193_482c_849f_e098d9811f9f.slice/crio-32e0a45376b6a48c5bb1546e8d9a0a401fef1a6b4377f4a6ffb8201045f613ec WatchSource:0}: Error finding container 32e0a45376b6a48c5bb1546e8d9a0a401fef1a6b4377f4a6ffb8201045f613ec: Status 404 returned error can't find the container with id 32e0a45376b6a48c5bb1546e8d9a0a401fef1a6b4377f4a6ffb8201045f613ec
Nov 02 14:00:49.939893 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:49.939856    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv" event={"ID":"139d0139-2193-482c-849f-e098d9811f9f","Type":"ContainerStarted","Data":"6739ad812f41873f31cc2936821256e7a3146b60ad148a9a2c59774befd275f9"}
Nov 02 14:00:49.940294 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:49.939911    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv" event={"ID":"139d0139-2193-482c-849f-e098d9811f9f","Type":"ContainerStarted","Data":"32e0a45376b6a48c5bb1546e8d9a0a401fef1a6b4377f4a6ffb8201045f613ec"}
Nov 02 14:00:50.062397 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:50.062299    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv" podStartSLOduration=2.062286778 podStartE2EDuration="2.062286778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:50.060873744 +0000 UTC m=+7478.624582155" watchObservedRunningTime="2026-11-02 14:00:50.062286778 +0000 UTC m=+7478.625995186"
Nov 02 14:00:50.947310 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:50.947277    4464 generic.go:356] "Generic (PLEG): container finished" podID="139d0139-2193-482c-849f-e098d9811f9f" containerID="6739ad812f41873f31cc2936821256e7a3146b60ad148a9a2c59774befd275f9" exitCode=0
Nov 02 14:00:50.947705 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:50.947357    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv" event={"ID":"139d0139-2193-482c-849f-e098d9811f9f","Type":"ContainerDied","Data":"6739ad812f41873f31cc2936821256e7a3146b60ad148a9a2c59774befd275f9"}
Nov 02 14:00:52.167440 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:52.167408    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv"
Nov 02 14:00:52.325878 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:52.325824    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/139d0139-2193-482c-849f-e098d9811f9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139d0139-2193-482c-849f-e098d9811f9f-maas-api-xks-ca\") pod \"139d0139-2193-482c-849f-e098d9811f9f\" (UID: \"139d0139-2193-482c-849f-e098d9811f9f\") "
Nov 02 14:00:52.326123 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:52.325910    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/139d0139-2193-482c-849f-e098d9811f9f-kube-api-access-8m9rq\" (UniqueName: \"kubernetes.io/projected/139d0139-2193-482c-849f-e098d9811f9f-kube-api-access-8m9rq\") pod \"139d0139-2193-482c-849f-e098d9811f9f\" (UID: \"139d0139-2193-482c-849f-e098d9811f9f\") "
Nov 02 14:00:52.326123 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:52.325992    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/139d0139-2193-482c-849f-e098d9811f9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139d0139-2193-482c-849f-e098d9811f9f-maas-api-openshift-ca\") pod \"139d0139-2193-482c-849f-e098d9811f9f\" (UID: \"139d0139-2193-482c-849f-e098d9811f9f\") "
Nov 02 14:00:52.326756 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:52.326714    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/139d0139-2193-482c-849f-e098d9811f9f-maas-api-openshift-ca" pod "139d0139-2193-482c-849f-e098d9811f9f" (UID: "139d0139-2193-482c-849f-e098d9811f9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:52.333146 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:52.333105    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139d0139-2193-482c-849f-e098d9811f9f-kube-api-access-8m9rq" pod "139d0139-2193-482c-849f-e098d9811f9f" (UID: "139d0139-2193-482c-849f-e098d9811f9f"). InnerVolumeSpecName "kube-api-access-8m9rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:52.340092 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:52.340055    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/139d0139-2193-482c-849f-e098d9811f9f-maas-api-xks-ca" pod "139d0139-2193-482c-849f-e098d9811f9f" (UID: "139d0139-2193-482c-849f-e098d9811f9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:52.427357 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:52.427319    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139d0139-2193-482c-849f-e098d9811f9f-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:00:52.427357 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:52.427360    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m9rq\" (UniqueName: \"kubernetes.io/projected/139d0139-2193-482c-849f-e098d9811f9f-kube-api-access-8m9rq\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:00:52.427643 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:52.427379    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139d0139-2193-482c-849f-e098d9811f9f-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:00:52.993000 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:52.992910    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv" event={"ID":"139d0139-2193-482c-849f-e098d9811f9f","Type":"ContainerDied","Data":"32e0a45376b6a48c5bb1546e8d9a0a401fef1a6b4377f4a6ffb8201045f613ec"}
Nov 02 14:00:52.993229 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:52.993024    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32e0a45376b6a48c5bb1546e8d9a0a401fef1a6b4377f4a6ffb8201045f613ec"
Nov 02 14:00:52.993229 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:52.993104    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b17e66-5kpkv"
Nov 02 14:00:55.169000 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:55.148370    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"]
Nov 02 14:00:55.234017 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:55.225186    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c62c279-f89tl"]
Nov 02 14:00:55.581069 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:55.581038    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f56d24d4-2246-43b9-b6fc-f18a73e91ba9" path="/var/lib/kubelet/pods/f56d24d4-2246-43b9-b6fc-f18a73e91ba9/volumes"
Nov 02 14:00:58.291458 ip-10-0-133-239 kubenswrapper[4464]: E1102 14:00:58.291421    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 14:00:59.518161 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.518129    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8"]
Nov 02 14:00:59.518608 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.518595    4464 cpu_manager.go:414] "containerMap: removing container" podUID="139d0139-2193-482c-849f-e098d9811f9f" containerName="revoke-keys"
Nov 02 14:00:59.518667 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.518609    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="139d0139-2193-482c-849f-e098d9811f9f" containerName="revoke-keys"
Nov 02 14:00:59.518709 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.518685    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="139d0139-2193-482c-849f-e098d9811f9f" containerName="revoke-keys"
Nov 02 14:00:59.522137 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.522121    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8"
Nov 02 14:00:59.573183 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.573149    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d394cf33-lttp8\" (UID: \"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8"
Nov 02 14:00:59.573322 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.573209    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d394cf33-lttp8\" (UID: \"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8"
Nov 02 14:00:59.573322 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.573301    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxl8d\" (UniqueName: \"kubernetes.io/projected/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-kube-api-access-kxl8d\") pod \"maas-api-revoke-sub-d394cf33-lttp8\" (UID: \"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8"
Nov 02 14:00:59.627512 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.627456    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:00:59.627512 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.627496    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 14:00:59.627512 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.627464    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:00:59.627741 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.627457    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:59.630104 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.630085    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8"]
Nov 02 14:00:59.674334 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.674308    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxl8d\" (UniqueName: \"kubernetes.io/projected/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-kube-api-access-kxl8d\") pod \"maas-api-revoke-sub-d394cf33-lttp8\" (UID: \"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8"
Nov 02 14:00:59.674520 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.674365    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d394cf33-lttp8\" (UID: \"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8"
Nov 02 14:00:59.674520 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.674411    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d394cf33-lttp8\" (UID: \"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8"
Nov 02 14:00:59.675255 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.675230    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d394cf33-lttp8\" (UID: \"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8"
Nov 02 14:00:59.676830 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.676802    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d394cf33-lttp8\" (UID: \"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8"
Nov 02 14:00:59.736225 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.736201    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxl8d\" (UniqueName: \"kubernetes.io/projected/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-kube-api-access-kxl8d\") pod \"maas-api-revoke-sub-d394cf33-lttp8\" (UID: \"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8"
Nov 02 14:00:59.835183 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:00:59.835111    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8"
Nov 02 14:01:00.212516 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:00.212488    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8"]
Nov 02 14:01:00.213252 ip-10-0-133-239 kubenswrapper[4464]: W1102 14:01:00.213228    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0bdbd9_dcdc_44d7_b035_1e215b3eae3c.slice/crio-f78e3ba9fe6f7455ebc8cca49a79d2f6668dcae3f3b258b962c61d4daf24872e WatchSource:0}: Error finding container f78e3ba9fe6f7455ebc8cca49a79d2f6668dcae3f3b258b962c61d4daf24872e: Status 404 returned error can't find the container with id f78e3ba9fe6f7455ebc8cca49a79d2f6668dcae3f3b258b962c61d4daf24872e
Nov 02 14:01:01.063673 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:01.063635    4464 generic.go:356] "Generic (PLEG): container finished" podID="5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c" containerID="da6e07f8f2f911b53dde253d90d3cfa80901d6e5390142626ab1b29081b87137" exitCode=0
Nov 02 14:01:01.064105 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:01.063721    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8" event={"ID":"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c","Type":"ContainerDied","Data":"da6e07f8f2f911b53dde253d90d3cfa80901d6e5390142626ab1b29081b87137"}
Nov 02 14:01:01.064105 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:01.063777    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8" event={"ID":"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c","Type":"ContainerStarted","Data":"f78e3ba9fe6f7455ebc8cca49a79d2f6668dcae3f3b258b962c61d4daf24872e"}
Nov 02 14:01:02.226377 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:02.226345    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8"
Nov 02 14:01:02.303279 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:02.303234    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-maas-api-xks-ca\") pod \"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c\" (UID: \"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c\") "
Nov 02 14:01:02.303497 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:02.303345    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-maas-api-openshift-ca\") pod \"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c\" (UID: \"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c\") "
Nov 02 14:01:02.303497 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:02.303471    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-kube-api-access-kxl8d\" (UniqueName: \"kubernetes.io/projected/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-kube-api-access-kxl8d\") pod \"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c\" (UID: \"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c\") "
Nov 02 14:01:02.304423 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:02.304391    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-maas-api-openshift-ca" pod "5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c" (UID: "5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:01:02.307012 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:02.306382    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-maas-api-xks-ca" pod "5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c" (UID: "5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:01:02.307012 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:02.306695    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-kube-api-access-kxl8d" pod "5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c" (UID: "5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c"). InnerVolumeSpecName "kube-api-access-kxl8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:02.405322 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:02.405225    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:01:02.405322 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:02.405267    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxl8d\" (UniqueName: \"kubernetes.io/projected/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-kube-api-access-kxl8d\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:01:02.405322 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:02.405283    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:01:03.080027 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:03.079993    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8"
Nov 02 14:01:03.080027 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:03.080013    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d394cf33-lttp8" event={"ID":"5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c","Type":"ContainerDied","Data":"f78e3ba9fe6f7455ebc8cca49a79d2f6668dcae3f3b258b962c61d4daf24872e"}
Nov 02 14:01:03.080305 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:03.080078    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f78e3ba9fe6f7455ebc8cca49a79d2f6668dcae3f3b258b962c61d4daf24872e"
Nov 02 14:01:04.886999 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:04.886094    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-dghm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:01:04.886999 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:04.886169    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-dghm7" podUID="2d10b658-29b5-4e07-84df-13326972bead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:05.041001 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:05.040075    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cnggl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:01:05.041001 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:05.040189    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cnggl" podUID="91491db1-c3aa-422e-9e3d-5575b4197aec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:05.076066 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:05.076031    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"]
Nov 02 14:01:05.092998 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:05.089836    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f87662f-zt2bf"]
Nov 02 14:01:05.596004 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:05.595803    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32f1f567-3513-45e6-a1fd-0ae80e4e628d" path="/var/lib/kubelet/pods/32f1f567-3513-45e6-a1fd-0ae80e4e628d/volumes"
Nov 02 14:01:10.268214 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.268178    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92"]
Nov 02 14:01:10.268627 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.268613    4464 cpu_manager.go:414] "containerMap: removing container" podUID="5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c" containerName="revoke-keys"
Nov 02 14:01:10.268673 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.268627    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c" containerName="revoke-keys"
Nov 02 14:01:10.268724 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.268715    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c" containerName="revoke-keys"
Nov 02 14:01:10.273019 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.273003    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92"
Nov 02 14:01:10.326407 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.326370    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:01:10.329999 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.329953    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92"]
Nov 02 14:01:10.354820 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.354787    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbgsd\" (UniqueName: \"kubernetes.io/projected/bf3dd18d-9948-4081-9c97-243b79b632a4-kube-api-access-vbgsd\") pod \"maas-api-revoke-sub-c7785d1f-khr92\" (UID: \"bf3dd18d-9948-4081-9c97-243b79b632a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92"
Nov 02 14:01:10.355008 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.354875    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3dd18d-9948-4081-9c97-243b79b632a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7785d1f-khr92\" (UID: \"bf3dd18d-9948-4081-9c97-243b79b632a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92"
Nov 02 14:01:10.355008 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.354961    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3dd18d-9948-4081-9c97-243b79b632a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7785d1f-khr92\" (UID: \"bf3dd18d-9948-4081-9c97-243b79b632a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92"
Nov 02 14:01:10.377243 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.377215    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:01:10.377243 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.377216    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 14:01:10.377413 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.377339    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:01:10.456150 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.456115    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3dd18d-9948-4081-9c97-243b79b632a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7785d1f-khr92\" (UID: \"bf3dd18d-9948-4081-9c97-243b79b632a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92"
Nov 02 14:01:10.456344 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.456182    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbgsd\" (UniqueName: \"kubernetes.io/projected/bf3dd18d-9948-4081-9c97-243b79b632a4-kube-api-access-vbgsd\") pod \"maas-api-revoke-sub-c7785d1f-khr92\" (UID: \"bf3dd18d-9948-4081-9c97-243b79b632a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92"
Nov 02 14:01:10.456344 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.456247    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3dd18d-9948-4081-9c97-243b79b632a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7785d1f-khr92\" (UID: \"bf3dd18d-9948-4081-9c97-243b79b632a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92"
Nov 02 14:01:10.456759 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.456735    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3dd18d-9948-4081-9c97-243b79b632a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7785d1f-khr92\" (UID: \"bf3dd18d-9948-4081-9c97-243b79b632a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92"
Nov 02 14:01:10.458594 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.458568    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3dd18d-9948-4081-9c97-243b79b632a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7785d1f-khr92\" (UID: \"bf3dd18d-9948-4081-9c97-243b79b632a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92"
Nov 02 14:01:10.518763 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.518691    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbgsd\" (UniqueName: \"kubernetes.io/projected/bf3dd18d-9948-4081-9c97-243b79b632a4-kube-api-access-vbgsd\") pod \"maas-api-revoke-sub-c7785d1f-khr92\" (UID: \"bf3dd18d-9948-4081-9c97-243b79b632a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92"
Nov 02 14:01:10.585785 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.585748    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92"
Nov 02 14:01:10.969614 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:10.969587    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92"]
Nov 02 14:01:10.970661 ip-10-0-133-239 kubenswrapper[4464]: W1102 14:01:10.970627    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3dd18d_9948_4081_9c97_243b79b632a4.slice/crio-608cfa4878e5a4177c658b77952c0e59406d39fb963878908d31924f0fd45827 WatchSource:0}: Error finding container 608cfa4878e5a4177c658b77952c0e59406d39fb963878908d31924f0fd45827: Status 404 returned error can't find the container with id 608cfa4878e5a4177c658b77952c0e59406d39fb963878908d31924f0fd45827
Nov 02 14:01:11.154445 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:11.154406    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92" event={"ID":"bf3dd18d-9948-4081-9c97-243b79b632a4","Type":"ContainerStarted","Data":"608cfa4878e5a4177c658b77952c0e59406d39fb963878908d31924f0fd45827"}
Nov 02 14:01:12.161294 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:12.161258    4464 generic.go:356] "Generic (PLEG): container finished" podID="bf3dd18d-9948-4081-9c97-243b79b632a4" containerID="7810e74d9a6ceb2bf0e6ce63122eb5845a58f3c3b971e59786e368a7c1e899b1" exitCode=0
Nov 02 14:01:12.161665 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:12.161352    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92" event={"ID":"bf3dd18d-9948-4081-9c97-243b79b632a4","Type":"ContainerDied","Data":"7810e74d9a6ceb2bf0e6ce63122eb5845a58f3c3b971e59786e368a7c1e899b1"}
Nov 02 14:01:12.522160 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:12.522127    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 14:01:12.538194 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:12.538171    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 14:01:12.582759 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:12.582735    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 14:01:12.599125 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:12.599100    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-w62gt_9d66d28c-8f94-4e4f-a07a-10b188c9bc6a/ovn-acl-logging/0.log"
Nov 02 14:01:13.363239 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:13.363199    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92"
Nov 02 14:01:13.489384 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:13.489345    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf3dd18d-9948-4081-9c97-243b79b632a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3dd18d-9948-4081-9c97-243b79b632a4-maas-api-openshift-ca\") pod \"bf3dd18d-9948-4081-9c97-243b79b632a4\" (UID: \"bf3dd18d-9948-4081-9c97-243b79b632a4\") "
Nov 02 14:01:13.489603 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:13.489425    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf3dd18d-9948-4081-9c97-243b79b632a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3dd18d-9948-4081-9c97-243b79b632a4-maas-api-xks-ca\") pod \"bf3dd18d-9948-4081-9c97-243b79b632a4\" (UID: \"bf3dd18d-9948-4081-9c97-243b79b632a4\") "
Nov 02 14:01:13.489680 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:13.489606    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf3dd18d-9948-4081-9c97-243b79b632a4-kube-api-access-vbgsd\" (UniqueName: \"kubernetes.io/projected/bf3dd18d-9948-4081-9c97-243b79b632a4-kube-api-access-vbgsd\") pod \"bf3dd18d-9948-4081-9c97-243b79b632a4\" (UID: \"bf3dd18d-9948-4081-9c97-243b79b632a4\") "
Nov 02 14:01:13.490635 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:13.490578    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3dd18d-9948-4081-9c97-243b79b632a4-maas-api-openshift-ca" pod "bf3dd18d-9948-4081-9c97-243b79b632a4" (UID: "bf3dd18d-9948-4081-9c97-243b79b632a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:01:13.505625 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:13.492779    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf3dd18d-9948-4081-9c97-243b79b632a4-maas-api-xks-ca" pod "bf3dd18d-9948-4081-9c97-243b79b632a4" (UID: "bf3dd18d-9948-4081-9c97-243b79b632a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:01:13.505625 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:13.492786    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3dd18d-9948-4081-9c97-243b79b632a4-kube-api-access-vbgsd" pod "bf3dd18d-9948-4081-9c97-243b79b632a4" (UID: "bf3dd18d-9948-4081-9c97-243b79b632a4"). InnerVolumeSpecName "kube-api-access-vbgsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:13.590864 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:13.590820    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbgsd\" (UniqueName: \"kubernetes.io/projected/bf3dd18d-9948-4081-9c97-243b79b632a4-kube-api-access-vbgsd\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:01:13.590864 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:13.590860    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3dd18d-9948-4081-9c97-243b79b632a4-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:01:13.591162 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:13.590876    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3dd18d-9948-4081-9c97-243b79b632a4-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:01:14.208000 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:14.203955    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92" event={"ID":"bf3dd18d-9948-4081-9c97-243b79b632a4","Type":"ContainerDied","Data":"608cfa4878e5a4177c658b77952c0e59406d39fb963878908d31924f0fd45827"}
Nov 02 14:01:14.208000 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:14.204060    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="608cfa4878e5a4177c658b77952c0e59406d39fb963878908d31924f0fd45827"
Nov 02 14:01:14.208000 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:14.204130    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7785d1f-khr92"
Nov 02 14:01:21.079938 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.079898    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js"]
Nov 02 14:01:21.080709 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.080687    4464 cpu_manager.go:414] "containerMap: removing container" podUID="bf3dd18d-9948-4081-9c97-243b79b632a4" containerName="revoke-keys"
Nov 02 14:01:21.080756 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.080713    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf3dd18d-9948-4081-9c97-243b79b632a4" containerName="revoke-keys"
Nov 02 14:01:21.080866 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.080852    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf3dd18d-9948-4081-9c97-243b79b632a4" containerName="revoke-keys"
Nov 02 14:01:21.086140 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.086114    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js"
Nov 02 14:01:21.141714 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.141681    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js"]
Nov 02 14:01:21.188236 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.188205    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:01:21.188236 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.188205    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:01:21.205947 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.205911    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8b6ffa2-bq6js\" (UID: \"6e355190-81c8-4ee0-b92a-7ae015c7a4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js"
Nov 02 14:01:21.206144 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.206054    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h26th\" (UniqueName: \"kubernetes.io/projected/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-kube-api-access-h26th\") pod \"maas-api-revoke-sub-f8b6ffa2-bq6js\" (UID: \"6e355190-81c8-4ee0-b92a-7ae015c7a4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js"
Nov 02 14:01:21.206144 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.206088    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8b6ffa2-bq6js\" (UID: \"6e355190-81c8-4ee0-b92a-7ae015c7a4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js"
Nov 02 14:01:21.212442 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.212415    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:01:21.212442 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.212437    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jcthm\""
Nov 02 14:01:21.307036 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.307003    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h26th\" (UniqueName: \"kubernetes.io/projected/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-kube-api-access-h26th\") pod \"maas-api-revoke-sub-f8b6ffa2-bq6js\" (UID: \"6e355190-81c8-4ee0-b92a-7ae015c7a4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js"
Nov 02 14:01:21.307276 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.307049    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8b6ffa2-bq6js\" (UID: \"6e355190-81c8-4ee0-b92a-7ae015c7a4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js"
Nov 02 14:01:21.307276 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.307129    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8b6ffa2-bq6js\" (UID: \"6e355190-81c8-4ee0-b92a-7ae015c7a4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js"
Nov 02 14:01:21.307835 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.307811    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8b6ffa2-bq6js\" (UID: \"6e355190-81c8-4ee0-b92a-7ae015c7a4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js"
Nov 02 14:01:21.309471 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.309451    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8b6ffa2-bq6js\" (UID: \"6e355190-81c8-4ee0-b92a-7ae015c7a4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js"
Nov 02 14:01:21.372734 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.372668    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h26th\" (UniqueName: \"kubernetes.io/projected/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-kube-api-access-h26th\") pod \"maas-api-revoke-sub-f8b6ffa2-bq6js\" (UID: \"6e355190-81c8-4ee0-b92a-7ae015c7a4b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js"
Nov 02 14:01:21.398785 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.398753    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js"
Nov 02 14:01:21.778447 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:21.778417    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js"]
Nov 02 14:01:21.779441 ip-10-0-133-239 kubenswrapper[4464]: W1102 14:01:21.779415    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e355190_81c8_4ee0_b92a_7ae015c7a4b6.slice/crio-68cefcbd48814bd2fccad7cb57f1aeecd11cbd9df93f20cabf6a64333c702465 WatchSource:0}: Error finding container 68cefcbd48814bd2fccad7cb57f1aeecd11cbd9df93f20cabf6a64333c702465: Status 404 returned error can't find the container with id 68cefcbd48814bd2fccad7cb57f1aeecd11cbd9df93f20cabf6a64333c702465
Nov 02 14:01:22.277013 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:22.276957    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js" event={"ID":"6e355190-81c8-4ee0-b92a-7ae015c7a4b6","Type":"ContainerStarted","Data":"c6f329e43b02c0e8cbec2a94e49d543c31fcc6a2a77f7796952da93e1a62b1cd"}
Nov 02 14:01:22.277497 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:22.277028    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js" event={"ID":"6e355190-81c8-4ee0-b92a-7ae015c7a4b6","Type":"ContainerStarted","Data":"68cefcbd48814bd2fccad7cb57f1aeecd11cbd9df93f20cabf6a64333c702465"}
Nov 02 14:01:22.397714 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:22.397654    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js" podStartSLOduration=1.397640236 podStartE2EDuration="1.397640236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:22.395963352 +0000 UTC m=+7510.959671790" watchObservedRunningTime="2026-11-02 14:01:22.397640236 +0000 UTC m=+7510.961348646"
Nov 02 14:01:23.283934 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:23.283894    4464 generic.go:356] "Generic (PLEG): container finished" podID="6e355190-81c8-4ee0-b92a-7ae015c7a4b6" containerID="c6f329e43b02c0e8cbec2a94e49d543c31fcc6a2a77f7796952da93e1a62b1cd" exitCode=0
Nov 02 14:01:23.284409 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:23.283999    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js" event={"ID":"6e355190-81c8-4ee0-b92a-7ae015c7a4b6","Type":"ContainerDied","Data":"c6f329e43b02c0e8cbec2a94e49d543c31fcc6a2a77f7796952da93e1a62b1cd"}
Nov 02 14:01:25.086334 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:25.086302    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js"
Nov 02 14:01:25.118676 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:25.118637    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-maas-api-openshift-ca\") pod \"6e355190-81c8-4ee0-b92a-7ae015c7a4b6\" (UID: \"6e355190-81c8-4ee0-b92a-7ae015c7a4b6\") "
Nov 02 14:01:25.119379 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:25.119356    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-maas-api-xks-ca\") pod \"6e355190-81c8-4ee0-b92a-7ae015c7a4b6\" (UID: \"6e355190-81c8-4ee0-b92a-7ae015c7a4b6\") "
Nov 02 14:01:25.119569 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:25.119549    4464 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-kube-api-access-h26th\" (UniqueName: \"kubernetes.io/projected/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-kube-api-access-h26th\") pod \"6e355190-81c8-4ee0-b92a-7ae015c7a4b6\" (UID: \"6e355190-81c8-4ee0-b92a-7ae015c7a4b6\") "
Nov 02 14:01:25.121225 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:25.119281    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-maas-api-openshift-ca" pod "6e355190-81c8-4ee0-b92a-7ae015c7a4b6" (UID: "6e355190-81c8-4ee0-b92a-7ae015c7a4b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:01:25.163392 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:25.163353    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-maas-api-xks-ca" pod "6e355190-81c8-4ee0-b92a-7ae015c7a4b6" (UID: "6e355190-81c8-4ee0-b92a-7ae015c7a4b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:01:25.164600 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:25.164573    4464 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-kube-api-access-h26th" pod "6e355190-81c8-4ee0-b92a-7ae015c7a4b6" (UID: "6e355190-81c8-4ee0-b92a-7ae015c7a4b6"). InnerVolumeSpecName "kube-api-access-h26th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:25.221734 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:25.221675    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-maas-api-openshift-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:01:25.222028 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:25.222003    4464 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-maas-api-xks-ca\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:01:25.222173 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:25.222157    4464 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h26th\" (UniqueName: \"kubernetes.io/projected/6e355190-81c8-4ee0-b92a-7ae015c7a4b6-kube-api-access-h26th\") on node \"ip-10-0-133-239.us-west-1.compute.internal\" DevicePath \"\""
Nov 02 14:01:25.450695 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:25.450593    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js" event={"ID":"6e355190-81c8-4ee0-b92a-7ae015c7a4b6","Type":"ContainerDied","Data":"68cefcbd48814bd2fccad7cb57f1aeecd11cbd9df93f20cabf6a64333c702465"}
Nov 02 14:01:25.450949 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:25.450931    4464 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68cefcbd48814bd2fccad7cb57f1aeecd11cbd9df93f20cabf6a64333c702465"
Nov 02 14:01:25.454138 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:25.451388    4464 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8b6ffa2-bq6js"
Nov 02 14:01:26.125139 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:26.125095    4464 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gt29z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:01:26.125775 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:26.125740    4464 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gt29z" podUID="bf09d973-3e5b-4daa-aa7f-cf839b61d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:28.291614 ip-10-0-133-239 kubenswrapper[4464]: E1102 14:01:28.291576    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 14:01:33.915364 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:33.915326    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-v6wvs"]
Nov 02 14:01:40.058709 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:40.058677    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"]
Nov 02 14:01:40.109603 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:40.109566    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9bae84-x88jj"]
Nov 02 14:01:41.590148 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:01:41.590115    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc37496-e0cf-4914-abb3-8353a81d4b13" path="/var/lib/kubelet/pods/6fc37496-e0cf-4914-abb3-8353a81d4b13/volumes"
Nov 02 14:01:58.292306 ip-10-0-133-239 kubenswrapper[4464]: E1102 14:01:58.292269    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 14:02:02.499617 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:02.499586    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-dcd84bcbc-k6pdq_17f4db95-d03a-4200-8b01-d7760251e7fa/manager/0.log"
Nov 02 14:02:02.899573 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:02.899486    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-28669_5d927c4f-316f-441e-bf28-dfbae4d88bfd/manager/0.log"
Nov 02 14:02:03.753914 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:03.753881    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5975b94bc-gwv4h_71cc9fa0-4ec8-499c-a857-9e7dffe12e63/manager/0.log"
Nov 02 14:02:04.361597 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:04.361567    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-6sbtx_5c5948cf-a3f1-4f12-b4ac-073db02a1911/manager/0.log"
Nov 02 14:02:04.633925 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:04.633853    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4z4sj_98f4686d-b894-4e32-91bb-479a67c05b1d/proxy/0.log"
Nov 02 14:02:06.198230 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:06.198200    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2c36f452-txs7l_ffe60823-6047-45b5-bf35-5f31620d27b3/revoke-keys/0.log"
Nov 02 14:02:06.445710 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:06.445679    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-39618c6d-czpvz_f7b69b23-2369-42cf-abd1-9103d400dba3/revoke-keys/0.log"
Nov 02 14:02:06.674561 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:06.674526    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3dedca5d-gjg9v_1c4523a7-535b-42ba-a578-1b1b80d8b0e4/revoke-keys/0.log"
Nov 02 14:02:06.900795 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:06.900769    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e7fb344-nssdp_93cafeac-cb97-4386-834b-a5602a4279dc/revoke-keys/0.log"
Nov 02 14:02:07.116341 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:07.116313    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5008c1e7-zn2ln_da77def7-0134-478f-90bc-6e08b479798f/revoke-keys/0.log"
Nov 02 14:02:07.334459 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:07.334422    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-70600120-l99v4_96512c88-7b5c-442f-a798-9a0bd0995c1c/revoke-keys/0.log"
Nov 02 14:02:07.559464 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:07.559434    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8ba75117-r5sgp_38ac09da-26a6-44f0-8d1e-a72581e3c468/revoke-keys/0.log"
Nov 02 14:02:07.795691 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:07.795666    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c7785d1f-khr92_bf3dd18d-9948-4081-9c97-243b79b632a4/revoke-keys/0.log"
Nov 02 14:02:08.010363 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:08.010335    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c8407b9d-2slkw_550a9c40-a0c5-4940-8680-83c6b02075c8/revoke-keys/0.log"
Nov 02 14:02:08.247568 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:08.247539    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cc0b8ec5-p8pgn_e23005db-6dc2-4e98-b839-0963d0e5a8cd/revoke-keys/0.log"
Nov 02 14:02:08.470413 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:08.470384    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cfdf614b-g47mv_aacadd1d-2562-47c4-9cde-ce2c1066ca49/revoke-keys/0.log"
Nov 02 14:02:08.690380 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:08.690355    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d394cf33-lttp8_5f0bdbd9-dcdc-44d7-b035-1e215b3eae3c/revoke-keys/0.log"
Nov 02 14:02:08.924887 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:08.924774    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d89b7fa8-t2x99_7adbd0a3-bbe9-4922-81c0-af50e8432e0b/revoke-keys/0.log"
Nov 02 14:02:09.149305 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:09.149275    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-db4acbc8-bvpmj_df181c99-e72f-4b8e-9a08-e416dbfb8837/revoke-keys/0.log"
Nov 02 14:02:09.367739 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:09.367711    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e8b17e66-5kpkv_139d0139-2193-482c-849f-e098d9811f9f/revoke-keys/0.log"
Nov 02 14:02:09.594451 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:09.594423    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f8b6ffa2-bq6js_6e355190-81c8-4ee0-b92a-7ae015c7a4b6/revoke-keys/0.log"
Nov 02 14:02:13.281210 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:13.281172    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-v6wvs_89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41/limitador/0.log"
Nov 02 14:02:14.806370 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:14.806341    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-s2tz8_c036f63f-f1c7-4d49-be9e-9f8c60907554/discovery/0.log"
Nov 02 14:02:15.052442 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:15.052412    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85ccdd577f-4jv5l_a85197ff-7b08-4de8-967a-d0f3a381b53b/kube-auth-proxy/0.log"
Nov 02 14:02:15.698905 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:15.698877    4464 scope.go:122] "RemoveContainer" containerID="983ca65b69a98edbb54cbebd7ace80c86ef190d23f328a01cd4567217a2bf8f3"
Nov 02 14:02:15.707520 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:15.707497    4464 scope.go:122] "RemoveContainer" containerID="e9a112d8e094d278cd45f13986570eede3b64df6fb9c7e6b8d207a10b8080e68"
Nov 02 14:02:16.159183 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:16.159152    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-cnggl_91491db1-c3aa-422e-9e3d-5575b4197aec/istio-proxy/0.log"
Nov 02 14:02:16.385065 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:16.385003    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-dghm7_2d10b658-29b5-4e07-84df-13326972bead/istio-proxy/0.log"
Nov 02 14:02:16.601343 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:16.601314    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-gt29z_bf09d973-3e5b-4daa-aa7f-cf839b61d1df/istio-proxy/0.log"
Nov 02 14:02:17.053875 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:17.053849    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-pk25x_6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116/istio-proxy/0.log"
Nov 02 14:02:17.942873 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:17.942842    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-69b46c5574-d6tqb_cd28a3fe-11b1-491e-b7f7-c10276803a4f/payload-pre-processing/0.log"
Nov 02 14:02:18.172683 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:18.172656    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-656dd59d89-ml7g4_2e12c7b1-dc24-4d4a-9cf4-682f4e82434d/payload-processing/0.log"
Nov 02 14:02:20.503281 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:20.503249    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-4l7p7_6a1bdd91-9d36-4c44-a3ac-694afddf8ba0/main/0.log"
Nov 02 14:02:20.726469 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:20.726441    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-8j57g_53517c94-5480-4ee7-abcf-cdc03f8af0a1/main/0.log"
Nov 02 14:02:20.944243 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:20.944170    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-r8qcw_55c46ddc-9659-48d9-92c6-d8363f573813/main/0.log"
Nov 02 14:02:21.659275 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:21.659241    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/1.log"
Nov 02 14:02:22.767436 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:22.767410    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-8bkgg_c1851d27-686d-4b50-bbf6-db2e372dd09c/main/0.log"
Nov 02 14:02:26.051072 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:26.051037    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"]
Nov 02 14:02:26.101419 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:26.101382    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc0b8ec5-p8pgn"]
Nov 02 14:02:27.581337 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:27.581304    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23005db-6dc2-4e98-b839-0963d0e5a8cd" path="/var/lib/kubelet/pods/e23005db-6dc2-4e98-b839-0963d0e5a8cd/volumes"
Nov 02 14:02:28.290391 ip-10-0-133-239 kubenswrapper[4464]: E1102 14:02:28.290359    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 14:02:33.415288 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:33.415250    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-685jh/must-gather-l9g7b"]
Nov 02 14:02:33.415782 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:33.415693    4464 cpu_manager.go:414] "containerMap: removing container" podUID="6e355190-81c8-4ee0-b92a-7ae015c7a4b6" containerName="revoke-keys"
Nov 02 14:02:33.415782 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:33.415704    4464 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e355190-81c8-4ee0-b92a-7ae015c7a4b6" containerName="revoke-keys"
Nov 02 14:02:33.415782 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:33.415769    4464 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e355190-81c8-4ee0-b92a-7ae015c7a4b6" containerName="revoke-keys"
Nov 02 14:02:33.420417 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:33.420398    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-685jh/must-gather-l9g7b"
Nov 02 14:02:33.478366 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:33.478339    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-685jh/must-gather-l9g7b"]
Nov 02 14:02:33.491408 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:33.491385    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rs84\" (UniqueName: \"kubernetes.io/projected/c3930275-50bf-4287-badf-af16569f39d9-kube-api-access-5rs84\") pod \"must-gather-l9g7b\" (UID: \"c3930275-50bf-4287-badf-af16569f39d9\") " pod="openshift-must-gather-685jh/must-gather-l9g7b"
Nov 02 14:02:33.491535 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:33.491453    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c3930275-50bf-4287-badf-af16569f39d9-must-gather-output\") pod \"must-gather-l9g7b\" (UID: \"c3930275-50bf-4287-badf-af16569f39d9\") " pod="openshift-must-gather-685jh/must-gather-l9g7b"
Nov 02 14:02:33.525825 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:33.525798    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-685jh\"/\"default-dockercfg-mjvj9\""
Nov 02 14:02:33.525825 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:33.525796    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-685jh\"/\"openshift-service-ca.crt\""
Nov 02 14:02:33.526008 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:33.525843    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-685jh\"/\"kube-root-ca.crt\""
Nov 02 14:02:33.592738 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:33.592712    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rs84\" (UniqueName: \"kubernetes.io/projected/c3930275-50bf-4287-badf-af16569f39d9-kube-api-access-5rs84\") pod \"must-gather-l9g7b\" (UID: \"c3930275-50bf-4287-badf-af16569f39d9\") " pod="openshift-must-gather-685jh/must-gather-l9g7b"
Nov 02 14:02:33.592891 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:33.592781    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c3930275-50bf-4287-badf-af16569f39d9-must-gather-output\") pod \"must-gather-l9g7b\" (UID: \"c3930275-50bf-4287-badf-af16569f39d9\") " pod="openshift-must-gather-685jh/must-gather-l9g7b"
Nov 02 14:02:33.593130 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:33.593111    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c3930275-50bf-4287-badf-af16569f39d9-must-gather-output\") pod \"must-gather-l9g7b\" (UID: \"c3930275-50bf-4287-badf-af16569f39d9\") " pod="openshift-must-gather-685jh/must-gather-l9g7b"
Nov 02 14:02:33.658813 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:33.658786    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rs84\" (UniqueName: \"kubernetes.io/projected/c3930275-50bf-4287-badf-af16569f39d9-kube-api-access-5rs84\") pod \"must-gather-l9g7b\" (UID: \"c3930275-50bf-4287-badf-af16569f39d9\") " pod="openshift-must-gather-685jh/must-gather-l9g7b"
Nov 02 14:02:33.749777 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:33.749753    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-685jh/must-gather-l9g7b"
Nov 02 14:02:34.143040 ip-10-0-133-239 kubenswrapper[4464]: W1102 14:02:34.143011    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3930275_50bf_4287_badf_af16569f39d9.slice/crio-2f1000b74b882ccec8c29c53fc9c3b3e5cc6672733b4addb373a0a7bcc6077e7 WatchSource:0}: Error finding container 2f1000b74b882ccec8c29c53fc9c3b3e5cc6672733b4addb373a0a7bcc6077e7: Status 404 returned error can't find the container with id 2f1000b74b882ccec8c29c53fc9c3b3e5cc6672733b4addb373a0a7bcc6077e7
Nov 02 14:02:34.143175 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:34.143078    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-685jh/must-gather-l9g7b"]
Nov 02 14:02:34.144838 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:34.144823    4464 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:02:34.877692 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:34.877652    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-685jh/must-gather-l9g7b" event={"ID":"c3930275-50bf-4287-badf-af16569f39d9","Type":"ContainerStarted","Data":"2f1000b74b882ccec8c29c53fc9c3b3e5cc6672733b4addb373a0a7bcc6077e7"}
Nov 02 14:02:36.061893 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:36.061860    4464 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"]
Nov 02 14:02:36.112080 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:36.112048    4464 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e7fb344-nssdp"]
Nov 02 14:02:37.584472 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:37.584437    4464 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93cafeac-cb97-4386-834b-a5602a4279dc" path="/var/lib/kubelet/pods/93cafeac-cb97-4386-834b-a5602a4279dc/volumes"
Nov 02 14:02:41.922574 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:41.922525    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-685jh/must-gather-l9g7b" event={"ID":"c3930275-50bf-4287-badf-af16569f39d9","Type":"ContainerStarted","Data":"7a18331c6450855ff081c962e9e3fabf9479bc98cbdfc586be20b1dd51ea076d"}
Nov 02 14:02:41.923065 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:41.922583    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-685jh/must-gather-l9g7b" event={"ID":"c3930275-50bf-4287-badf-af16569f39d9","Type":"ContainerStarted","Data":"c2466628bfdc6164ad7fb7321989e63fd9baa5722ce9e4979f6f09b2b3f87168"}
Nov 02 14:02:42.040869 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:42.040813    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-685jh/must-gather-l9g7b" podStartSLOduration=1.7933569870000001 podStartE2EDuration="9.040796862s" totalImagesPullingTime="7.247439875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:42.039576848 +0000 UTC m=+7590.603285275" watchObservedRunningTime="2026-11-02 14:02:42.040796862 +0000 UTC m=+7590.604505270"
Nov 02 14:02:42.819235 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:42.819200    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sw29m_8161b19f-1a15-4f7d-9232-a9248995abf6/global-pull-secret-syncer/0.log"
Nov 02 14:02:45.487550 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:45.487513    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-dcd84bcbc-k6pdq_17f4db95-d03a-4200-8b01-d7760251e7fa/manager/0.log"
Nov 02 14:02:45.695286 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:45.695256    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-28669_5d927c4f-316f-441e-bf28-dfbae4d88bfd/manager/0.log"
Nov 02 14:02:46.240936 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:46.240904    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k6kl8_b7058574-e988-4cc5-8a0c-fa961b9080c1/konnectivity-agent/0.log"
Nov 02 14:02:46.621880 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:46.621796    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5975b94bc-gwv4h_71cc9fa0-4ec8-499c-a857-9e7dffe12e63/manager/0.log"
Nov 02 14:02:46.661136 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:46.661103    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-239.us-west-1.compute.internal_1797323b44d1a789ea358554c2680a4d/haproxy/0.log"
Nov 02 14:02:47.560233 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:47.560199    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-6sbtx_5c5948cf-a3f1-4f12-b4ac-073db02a1911/manager/0.log"
Nov 02 14:02:47.678927 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:47.678893    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4z4sj_98f4686d-b894-4e32-91bb-479a67c05b1d/proxy/0.log"
Nov 02 14:02:50.634383 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:50.634348    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-dcd84bcbc-k6pdq_17f4db95-d03a-4200-8b01-d7760251e7fa/manager/0.log"
Nov 02 14:02:50.834348 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:50.834321    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-28669_5d927c4f-316f-441e-bf28-dfbae4d88bfd/manager/0.log"
Nov 02 14:02:51.753488 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:51.753445    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5975b94bc-gwv4h_71cc9fa0-4ec8-499c-a857-9e7dffe12e63/manager/0.log"
Nov 02 14:02:52.845968 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:52.845933    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-6sbtx_5c5948cf-a3f1-4f12-b4ac-073db02a1911/manager/0.log"
Nov 02 14:02:52.846817 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:52.846792    4464 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"]
Nov 02 14:02:52.853472 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:52.853447    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:52.908512 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:52.908483    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"]
Nov 02 14:02:52.956012 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:52.955965    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z927c\"/\"kube-root-ca.crt\""
Nov 02 14:02:52.956012 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:52.955999    4464 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z927c\"/\"default-dockercfg-b99r6\""
Nov 02 14:02:52.956012 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:52.955965    4464 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z927c\"/\"openshift-service-ca.crt\""
Nov 02 14:02:52.993665 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:52.993626    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0-podres\") pod \"perf-node-gather-daemonset-jfzjj\" (UID: \"1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:52.993846 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:52.993671    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l56kr\" (UniqueName: \"kubernetes.io/projected/1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0-kube-api-access-l56kr\") pod \"perf-node-gather-daemonset-jfzjj\" (UID: \"1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:52.993846 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:52.993786    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0-sys\") pod \"perf-node-gather-daemonset-jfzjj\" (UID: \"1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:52.993846 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:52.993809    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0-proc\") pod \"perf-node-gather-daemonset-jfzjj\" (UID: \"1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:52.994054 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:52.993915    4464 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0-lib-modules\") pod \"perf-node-gather-daemonset-jfzjj\" (UID: \"1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:53.044681 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:53.044655    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4z4sj_98f4686d-b894-4e32-91bb-479a67c05b1d/proxy/0.log"
Nov 02 14:02:53.095595 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:53.095557    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0-podres\") pod \"perf-node-gather-daemonset-jfzjj\" (UID: \"1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:53.095755 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:53.095600    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l56kr\" (UniqueName: \"kubernetes.io/projected/1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0-kube-api-access-l56kr\") pod \"perf-node-gather-daemonset-jfzjj\" (UID: \"1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:53.095755 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:53.095688    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0-podres\") pod \"perf-node-gather-daemonset-jfzjj\" (UID: \"1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:53.095755 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:53.095713    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0-sys\") pod \"perf-node-gather-daemonset-jfzjj\" (UID: \"1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:53.095755 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:53.095738    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0-proc\") pod \"perf-node-gather-daemonset-jfzjj\" (UID: \"1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:53.095755 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:53.095743    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0-sys\") pod \"perf-node-gather-daemonset-jfzjj\" (UID: \"1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:53.095928 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:53.095790    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0-proc\") pod \"perf-node-gather-daemonset-jfzjj\" (UID: \"1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:53.095928 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:53.095796    4464 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0-lib-modules\") pod \"perf-node-gather-daemonset-jfzjj\" (UID: \"1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:53.095928 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:53.095838    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0-lib-modules\") pod \"perf-node-gather-daemonset-jfzjj\" (UID: \"1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:53.157872 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:53.157796    4464 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l56kr\" (UniqueName: \"kubernetes.io/projected/1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0-kube-api-access-l56kr\") pod \"perf-node-gather-daemonset-jfzjj\" (UID: \"1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:53.167769 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:53.167745    4464 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:53.547886 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:53.547858    4464 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"]
Nov 02 14:02:53.549066 ip-10-0-133-239 kubenswrapper[4464]: W1102 14:02:53.549040    4464 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1860d70c_b9b2_47a7_a9c9_2aa48a37c3e0.slice/crio-652e785ad5e135a6154fe4914aca498a1290f0c7b198dac6fad18f66d6370c90 WatchSource:0}: Error finding container 652e785ad5e135a6154fe4914aca498a1290f0c7b198dac6fad18f66d6370c90: Status 404 returned error can't find the container with id 652e785ad5e135a6154fe4914aca498a1290f0c7b198dac6fad18f66d6370c90
Nov 02 14:02:54.007034 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:54.006969    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj" event={"ID":"1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0","Type":"ContainerStarted","Data":"946300fb6e7fc0ca886faeac0a235df204dbb8b59cc917f2d0d7f5663535bed1"}
Nov 02 14:02:54.007546 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:54.007053    4464 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj" event={"ID":"1860d70c-b9b2-47a7-a9c9-2aa48a37c3e0","Type":"ContainerStarted","Data":"652e785ad5e135a6154fe4914aca498a1290f0c7b198dac6fad18f66d6370c90"}
Nov 02 14:02:54.007546 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:54.007121    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:02:54.126335 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:54.126281    4464 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj" podStartSLOduration=2.126267382 podStartE2EDuration="2.126267382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:54.12449459 +0000 UTC m=+7602.688203001" watchObservedRunningTime="2026-11-02 14:02:54.126267382 +0000 UTC m=+7602.689975788"
Nov 02 14:02:55.079400 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:55.079307    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-v6wvs_89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41/limitador/0.log"
Nov 02 14:02:56.065801 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:56.065756    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-dcd84bcbc-k6pdq_17f4db95-d03a-4200-8b01-d7760251e7fa/manager/0.log"
Nov 02 14:02:56.266952 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:56.266902    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-28669_5d927c4f-316f-441e-bf28-dfbae4d88bfd/manager/0.log"
Nov 02 14:02:57.464367 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:57.464326    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5975b94bc-gwv4h_71cc9fa0-4ec8-499c-a857-9e7dffe12e63/manager/0.log"
Nov 02 14:02:58.202839 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:58.202793    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_84a78b98-6493-41d0-9d0a-692504bd16c7/alertmanager/0.log"
Nov 02 14:02:58.299164 ip-10-0-133-239 kubenswrapper[4464]: E1102 14:02:58.299129    4464 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-t692p" containerName="extractor"
Nov 02 14:02:58.442290 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:58.442253    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-6sbtx_5c5948cf-a3f1-4f12-b4ac-073db02a1911/manager/0.log"
Nov 02 14:02:58.618863 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:58.618832    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_84a78b98-6493-41d0-9d0a-692504bd16c7/config-reloader/0.log"
Nov 02 14:02:58.866317 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:58.866291    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4z4sj_98f4686d-b894-4e32-91bb-479a67c05b1d/proxy/0.log"
Nov 02 14:02:59.035232 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:59.035186    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_84a78b98-6493-41d0-9d0a-692504bd16c7/kube-rbac-proxy-web/0.log"
Nov 02 14:02:59.449377 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:59.449262    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_84a78b98-6493-41d0-9d0a-692504bd16c7/kube-rbac-proxy/0.log"
Nov 02 14:02:59.865417 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:02:59.865388    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_84a78b98-6493-41d0-9d0a-692504bd16c7/kube-rbac-proxy-metric/0.log"
Nov 02 14:03:00.024288 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:00.024261    4464 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z927c/perf-node-gather-daemonset-jfzjj"
Nov 02 14:03:00.287755 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:00.287728    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_84a78b98-6493-41d0-9d0a-692504bd16c7/prom-label-proxy/0.log"
Nov 02 14:03:00.713280 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:00.713188    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_84a78b98-6493-41d0-9d0a-692504bd16c7/init-config-reloader/0.log"
Nov 02 14:03:01.556558 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:01.556532    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7ncxv_bcd1b647-47f8-4785-bfb6-acfa67de94fe/kube-state-metrics/0.log"
Nov 02 14:03:01.968387 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:01.968364    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7ncxv_bcd1b647-47f8-4785-bfb6-acfa67de94fe/kube-rbac-proxy-main/0.log"
Nov 02 14:03:02.383042 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:02.382955    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7ncxv_bcd1b647-47f8-4785-bfb6-acfa67de94fe/kube-rbac-proxy-self/0.log"
Nov 02 14:03:02.804641 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:02.804610    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-65867d596c-9knn9_cc07f22e-8721-4fb4-bd15-ce10bb6bc7d3/metrics-server/0.log"
Nov 02 14:03:03.228965 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:03.228939    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-rqs2c_7c3b3675-ec09-40d1-a2e9-0a74dcb48286/monitoring-plugin/0.log"
Nov 02 14:03:04.924580 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:04.924547    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ccmmd_9fffe20a-88c8-4196-8b56-5ddf2c71fcaf/node-exporter/0.log"
Nov 02 14:03:05.141520 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:05.141485    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-s2tz8_c036f63f-f1c7-4d49-be9e-9f8c60907554/discovery/0.log"
Nov 02 14:03:05.250170 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:05.250145    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85ccdd577f-4jv5l_a85197ff-7b08-4de8-967a-d0f3a381b53b/kube-auth-proxy/0.log"
Nov 02 14:03:05.352395 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:05.352367    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ccmmd_9fffe20a-88c8-4196-8b56-5ddf2c71fcaf/kube-rbac-proxy/0.log"
Nov 02 14:03:05.770037 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:05.770009    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ccmmd_9fffe20a-88c8-4196-8b56-5ddf2c71fcaf/init-textfile/0.log"
Nov 02 14:03:06.335078 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:06.335051    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-cnggl_91491db1-c3aa-422e-9e3d-5575b4197aec/istio-proxy/0.log"
Nov 02 14:03:06.552860 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:06.552832    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-dghm7_2d10b658-29b5-4e07-84df-13326972bead/istio-proxy/0.log"
Nov 02 14:03:06.772208 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:06.772158    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-gt29z_bf09d973-3e5b-4daa-aa7f-cf839b61d1df/istio-proxy/0.log"
Nov 02 14:03:07.202343 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:07.202268    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-pk25x_6c163eda-a3a0-4ea5-a8ea-ccc9f1d15116/istio-proxy/0.log"
Nov 02 14:03:08.092779 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:08.092747    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-69b46c5574-d6tqb_cd28a3fe-11b1-491e-b7f7-c10276803a4f/payload-pre-processing/0.log"
Nov 02 14:03:08.308749 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:08.308720    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-656dd59d89-ml7g4_2e12c7b1-dc24-4d4a-9cf4-682f4e82434d/payload-processing/0.log"
Nov 02 14:03:13.001506 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:13.001453    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cdf8c8bbd-b64wq_1d48ddc2-9b07-448a-b214-9aadd38d54b4/telemeter-client/0.log"
Nov 02 14:03:13.416406 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:13.416317    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cdf8c8bbd-b64wq_1d48ddc2-9b07-448a-b214-9aadd38d54b4/reload/0.log"
Nov 02 14:03:13.773183 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:13.773155    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-v6wvs_89ab4a35-50b7-46a3-b5a1-6f43ed3b9a41/limitador/0.log"
Nov 02 14:03:13.832231 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:13.832180    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cdf8c8bbd-b64wq_1d48ddc2-9b07-448a-b214-9aadd38d54b4/kube-rbac-proxy/0.log"
Nov 02 14:03:14.252550 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:14.252524    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4bb8cbdb-fh5rn_06d0e7da-fbf7-4c6b-b532-9ef01ab60150/thanos-query/0.log"
Nov 02 14:03:14.668192 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:14.668102    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4bb8cbdb-fh5rn_06d0e7da-fbf7-4c6b-b532-9ef01ab60150/kube-rbac-proxy-web/0.log"
Nov 02 14:03:15.084192 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:15.084148    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4bb8cbdb-fh5rn_06d0e7da-fbf7-4c6b-b532-9ef01ab60150/kube-rbac-proxy/0.log"
Nov 02 14:03:15.501397 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:15.501371    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4bb8cbdb-fh5rn_06d0e7da-fbf7-4c6b-b532-9ef01ab60150/prom-label-proxy/0.log"
Nov 02 14:03:15.732585 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:15.732550    4464 scope.go:122] "RemoveContainer" containerID="9855d2984ba2c05c56f948fef20cd963e543f389f02d97c868349d3296f867ab"
Nov 02 14:03:15.921247 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:15.921169    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4bb8cbdb-fh5rn_06d0e7da-fbf7-4c6b-b532-9ef01ab60150/kube-rbac-proxy-rules/0.log"
Nov 02 14:03:16.348179 ip-10-0-133-239 kubenswrapper[4464]: I1102 14:03:16.348139    4464 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b4bb8cbdb-fh5rn_06d0e7da-fbf7-4c6b-b532-9ef01ab60150/kube-rbac-proxy-metrics/0.log"